2023-03-30 16:53:28,449 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea 2023-03-30 16:53:28,463 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-03-30 16:53:28,478 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-03-30 16:53:28,479 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/cluster_6981959d-287c-2be4-2da0-db5747864972, deleteOnExit=true 2023-03-30 16:53:28,479 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-03-30 16:53:28,480 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/test.cache.data in system properties and HBase conf 2023-03-30 16:53:28,480 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/hadoop.tmp.dir in system properties and HBase conf 2023-03-30 16:53:28,481 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/hadoop.log.dir in system properties and HBase conf 2023-03-30 16:53:28,481 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/mapreduce.cluster.local.dir in system properties and HBase conf 2023-03-30 16:53:28,482 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-03-30 16:53:28,482 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-03-30 16:53:28,606 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-30 16:53:29,036 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-03-30 16:53:29,040 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-03-30 16:53:29,041 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-03-30 16:53:29,041 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-03-30 16:53:29,041 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-30 16:53:29,042 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-03-30 16:53:29,042 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-03-30 16:53:29,043 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-30 16:53:29,043 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-30 16:53:29,043 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-03-30 16:53:29,044 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/nfs.dump.dir in system properties and HBase conf 2023-03-30 16:53:29,044 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/java.io.tmpdir in system properties and HBase conf 2023-03-30 16:53:29,045 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-30 16:53:29,045 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-03-30 16:53:29,046 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-03-30 16:53:29,677 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-30 16:53:29,681 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-30 16:53:30,006 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-03-30 16:53:30,190 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-03-30 16:53:30,208 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-30 16:53:30,246 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-03-30 16:53:30,277 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/java.io.tmpdir/Jetty_localhost_43471_hdfs____li77x5/webapp 2023-03-30 16:53:30,438 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43471 2023-03-30 16:53:30,479 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-30 16:53:30,479 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-30 16:53:30,905 WARN [Listener at localhost/46323] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-30 16:53:31,053 WARN [Listener at localhost/46323] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-30 16:53:31,071 WARN [Listener at localhost/46323] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-30 16:53:31,078 INFO [Listener at localhost/46323] log.Slf4jLog(67): jetty-6.1.26 2023-03-30 16:53:31,084 INFO [Listener at localhost/46323] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/java.io.tmpdir/Jetty_localhost_45415_datanode____.x02ixu/webapp 2023-03-30 16:53:31,187 INFO [Listener at localhost/46323] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45415 2023-03-30 16:53:31,604 WARN [Listener at localhost/38075] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-30 16:53:32,125 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x2bda9eb382f06063: Processing first storage report for DS-b8408660-09bd-410e-b9fa-5156fd0bd779 from datanode feae19bc-dd98-4b85-9889-0ea49bd6ef8b 2023-03-30 16:53:32,128 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x2bda9eb382f06063: from storage DS-b8408660-09bd-410e-b9fa-5156fd0bd779 node DatanodeRegistration(127.0.0.1:35955, datanodeUuid=feae19bc-dd98-4b85-9889-0ea49bd6ef8b, infoPort=41315, infoSecurePort=0, ipcPort=38075, storageInfo=lv=-57;cid=testClusterID;nsid=317086095;c=1680195209756), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-03-30 16:53:32,128 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x2bda9eb382f06063: Processing first storage report for DS-5fa026ac-c41d-41bb-9848-3a60a308442c from datanode feae19bc-dd98-4b85-9889-0ea49bd6ef8b 2023-03-30 16:53:32,128 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x2bda9eb382f06063: from storage DS-5fa026ac-c41d-41bb-9848-3a60a308442c node DatanodeRegistration(127.0.0.1:35955, datanodeUuid=feae19bc-dd98-4b85-9889-0ea49bd6ef8b, infoPort=41315, infoSecurePort=0, ipcPort=38075, storageInfo=lv=-57;cid=testClusterID;nsid=317086095;c=1680195209756), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-03-30 16:53:32,144 DEBUG [Listener at localhost/38075] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea 2023-03-30 16:53:32,218 INFO [Listener at localhost/38075] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/cluster_6981959d-287c-2be4-2da0-db5747864972/zookeeper_0, clientPort=58435, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/cluster_6981959d-287c-2be4-2da0-db5747864972/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/cluster_6981959d-287c-2be4-2da0-db5747864972/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-03-30 16:53:32,232 INFO [Listener at localhost/38075] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=58435 2023-03-30 16:53:32,240 INFO [Listener at localhost/38075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-30 16:53:32,242 INFO [Listener at localhost/38075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-30 16:53:32,922 INFO [Listener at localhost/38075] util.FSUtils(471): Created version file at hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c with version=8 2023-03-30 16:53:32,922 INFO [Listener at localhost/38075] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/hbase-staging 2023-03-30 16:53:33,246 INFO [Listener at localhost/38075] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-30 16:53:33,726 INFO [Listener at localhost/38075] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-03-30 16:53:33,763 INFO [Listener at localhost/38075] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-30 16:53:33,764 INFO [Listener at localhost/38075] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-30 16:53:33,764 INFO [Listener at localhost/38075] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-30 16:53:33,764 INFO [Listener at localhost/38075] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-30 16:53:33,764 INFO [Listener at localhost/38075] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-30 16:53:33,902 INFO [Listener at localhost/38075] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-03-30 16:53:33,973 DEBUG [Listener at localhost/38075] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-03-30 16:53:34,085 INFO [Listener at localhost/38075] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40565 2023-03-30 16:53:34,096 INFO [Listener at localhost/38075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-30 16:53:34,098 INFO [Listener at localhost/38075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-30 16:53:34,118 INFO [Listener at localhost/38075] zookeeper.RecoverableZooKeeper(93): Process identifier=master:40565 connecting to ZooKeeper ensemble=127.0.0.1:58435 2023-03-30 16:53:34,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:405650x0, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-30 16:53:34,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:40565-0x10136583d700000 connected 2023-03-30 16:53:34,187 DEBUG [Listener at localhost/38075] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-30 16:53:34,188 DEBUG [Listener at localhost/38075] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-30 16:53:34,191 DEBUG [Listener at localhost/38075] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-30 16:53:34,199 DEBUG [Listener at localhost/38075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40565 2023-03-30 16:53:34,199 DEBUG [Listener at localhost/38075] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40565 2023-03-30 16:53:34,200 DEBUG [Listener at localhost/38075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40565 2023-03-30 16:53:34,202 DEBUG [Listener at localhost/38075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40565 2023-03-30 16:53:34,204 DEBUG [Listener at localhost/38075] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40565 2023-03-30 16:53:34,210 INFO [Listener at localhost/38075] master.HMaster(444): hbase.rootdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c, hbase.cluster.distributed=false 2023-03-30 16:53:34,286 INFO [Listener at localhost/38075] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-03-30 16:53:34,286 INFO [Listener at localhost/38075] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-30 16:53:34,286 INFO [Listener at localhost/38075] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-30 16:53:34,286 INFO [Listener at localhost/38075] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-30 16:53:34,286 INFO [Listener at localhost/38075] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-30 16:53:34,286 INFO [Listener at localhost/38075] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-30 16:53:34,291 INFO [Listener at localhost/38075] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-30 16:53:34,294 INFO [Listener at localhost/38075] ipc.NettyRpcServer(120): Bind to /172.31.14.131:45193 2023-03-30 16:53:34,297 INFO [Listener at localhost/38075] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-30 16:53:34,303 DEBUG [Listener at localhost/38075] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-03-30 16:53:34,304 INFO [Listener at localhost/38075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-30 16:53:34,307 INFO [Listener at localhost/38075] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-30 16:53:34,309 INFO [Listener at localhost/38075] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:45193 connecting to ZooKeeper ensemble=127.0.0.1:58435 2023-03-30 16:53:34,313 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:451930x0, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-30 16:53:34,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:45193-0x10136583d700001 connected 2023-03-30 16:53:34,314 DEBUG [Listener at localhost/38075] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-30 16:53:34,315 DEBUG [Listener at localhost/38075] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-30 16:53:34,316 DEBUG [Listener at localhost/38075] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-30 16:53:34,317 DEBUG [Listener at localhost/38075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45193 2023-03-30 16:53:34,317 DEBUG [Listener at localhost/38075] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45193 2023-03-30 16:53:34,318 DEBUG [Listener at localhost/38075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45193 2023-03-30 16:53:34,318 DEBUG [Listener at localhost/38075] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45193 2023-03-30 16:53:34,319 DEBUG [Listener at localhost/38075] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45193 2023-03-30 16:53:34,327 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,40565,1680195213071 2023-03-30 16:53:34,338 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-30 16:53:34,339 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,40565,1680195213071 2023-03-30 16:53:34,358 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-30 16:53:34,358 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-30 16:53:34,358 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-30 16:53:34,359 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-30 16:53:34,360 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,40565,1680195213071 from backup master directory 2023-03-30 16:53:34,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-30 16:53:34,363 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,40565,1680195213071 2023-03-30 16:53:34,363 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-30 16:53:34,364 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-30 16:53:34,364 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,40565,1680195213071 2023-03-30 16:53:34,367 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-03-30 16:53:34,368 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-03-30 16:53:34,455 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/hbase.id with ID: 8c2feac2-0b95-444d-af72-c4c22fd85fbb 2023-03-30 16:53:34,498 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-30 16:53:34,512 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-30 16:53:34,550 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2869d048 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:53:34,583 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8b363ac, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:53:34,606 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-30 16:53:34,608 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-03-30 16:53:34,625 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-03-30 16:53:34,625 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-03-30 16:53:34,627 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:53:34,630 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:53:34,631 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-30 16:53:34,664 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/MasterData/data/master/store-tmp 2023-03-30 16:53:34,689 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:53:34,689 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-30 16:53:34,689 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-30 16:53:34,689 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-30 16:53:34,690 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-30 16:53:34,690 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-30 16:53:34,690 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-30 16:53:34,690 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-30 16:53:34,691 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/MasterData/WALs/jenkins-hbase4.apache.org,40565,1680195213071 2023-03-30 16:53:34,714 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40565%2C1680195213071, suffix=, logDir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/MasterData/WALs/jenkins-hbase4.apache.org,40565,1680195213071, archiveDir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/MasterData/oldWALs, maxLogs=10 2023-03-30 16:53:34,764 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35955,DS-b8408660-09bd-410e-b9fa-5156fd0bd779,DISK] 2023-03-30 16:53:34,773 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:53:34,831 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/MasterData/WALs/jenkins-hbase4.apache.org,40565,1680195213071/jenkins-hbase4.apache.org%2C40565%2C1680195213071.1680195214723 2023-03-30 16:53:34,831 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35955,DS-b8408660-09bd-410e-b9fa-5156fd0bd779,DISK]] 2023-03-30 16:53:34,832 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-03-30 16:53:34,832 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:53:34,835 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-03-30 16:53:34,836 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-03-30 16:53:34,890 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-03-30 16:53:34,896 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-03-30 16:53:34,919 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-03-30 16:53:34,931 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:53:34,936 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-30 16:53:34,937 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-30 16:53:34,958 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-03-30 16:53:34,962 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-30 16:53:34,964 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70265365, jitterRate=0.04703553020954132}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-30 16:53:34,964 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-30 16:53:34,965 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-03-30 16:53:34,986 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-03-30 16:53:34,986 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-03-30 16:53:34,989 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-03-30 16:53:34,991 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-03-30 16:53:35,025 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 33 msec 2023-03-30 16:53:35,025 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-03-30 16:53:35,053 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-03-30 16:53:35,060 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-03-30 16:53:35,091 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-03-30 16:53:35,094 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-30 16:53:35,097 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-03-30 16:53:35,102 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-03-30 16:53:35,106 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-03-30 16:53:35,110 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-30 16:53:35,111 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-03-30 16:53:35,112 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-03-30 16:53:35,129 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-03-30 16:53:35,136 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-30 16:53:35,136 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-30 16:53:35,136 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-30 16:53:35,137 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,40565,1680195213071, sessionid=0x10136583d700000, setting cluster-up flag (Was=false) 2023-03-30 16:53:35,152 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-30 16:53:35,159 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-03-30 16:53:35,161 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,40565,1680195213071 2023-03-30 16:53:35,166 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-30 16:53:35,172 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-03-30 16:53:35,173 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,40565,1680195213071 2023-03-30 16:53:35,176 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.hbase-snapshot/.tmp 2023-03-30 16:53:35,229 INFO [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(952): ClusterId : 8c2feac2-0b95-444d-af72-c4c22fd85fbb 2023-03-30 16:53:35,233 DEBUG [RS:0;jenkins-hbase4:45193] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-30 16:53:35,238 DEBUG [RS:0;jenkins-hbase4:45193] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-30 16:53:35,239 DEBUG [RS:0;jenkins-hbase4:45193] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-30 16:53:35,243 DEBUG [RS:0;jenkins-hbase4:45193] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-30 16:53:35,244 DEBUG [RS:0;jenkins-hbase4:45193] zookeeper.ReadOnlyZKClient(139): Connect 0x1e714ffa to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:53:35,249 DEBUG [RS:0;jenkins-hbase4:45193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10b6a244, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:53:35,250 DEBUG [RS:0;jenkins-hbase4:45193] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1489248a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-30 16:53:35,272 DEBUG [RS:0;jenkins-hbase4:45193] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:45193 2023-03-30 16:53:35,276 INFO [RS:0;jenkins-hbase4:45193] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-30 16:53:35,276 INFO [RS:0;jenkins-hbase4:45193] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-30 16:53:35,276 DEBUG [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(1023): About to register with Master. 2023-03-30 16:53:35,278 INFO [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,40565,1680195213071 with isa=jenkins-hbase4.apache.org/172.31.14.131:45193, startcode=1680195214285 2023-03-30 16:53:35,297 DEBUG [RS:0;jenkins-hbase4:45193] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-30 16:53:35,321 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-03-30 16:53:35,330 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-30 16:53:35,331 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-30 16:53:35,331 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-30 16:53:35,331 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-30 16:53:35,331 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-03-30 16:53:35,331 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-30 16:53:35,331 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-30 16:53:35,331 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-30 16:53:35,332 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680195245332 2023-03-30 16:53:35,334 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-03-30 16:53:35,338 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-03-30 16:53:35,339 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-03-30 16:53:35,344 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-30 16:53:35,346 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-03-30 16:53:35,354 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-03-30 16:53:35,355 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-03-30 16:53:35,355 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-03-30 16:53:35,355 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-03-30 16:53:35,356 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-30 16:53:35,357 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37263, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-03-30 16:53:35,358 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-03-30 16:53:35,359 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-03-30 16:53:35,360 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-03-30 16:53:35,364 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-03-30 16:53:35,364 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-03-30 16:53:35,366 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680195215366,5,FailOnTimeoutGroup] 2023-03-30 16:53:35,367 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680195215367,5,FailOnTimeoutGroup] 2023-03-30 16:53:35,367 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-30 16:53:35,367 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-03-30 16:53:35,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:35,368 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-03-30 16:53:35,368 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-03-30 16:53:35,397 DEBUG [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(2829): Master is not running yet 2023-03-30 16:53:35,398 WARN [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-03-30 16:53:35,499 INFO [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,40565,1680195213071 with isa=jenkins-hbase4.apache.org/172.31.14.131:45193, startcode=1680195214285 2023-03-30 16:53:35,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:35,510 DEBUG [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c 2023-03-30 16:53:35,510 DEBUG [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:46323 2023-03-30 16:53:35,510 DEBUG [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-30 16:53:35,515 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-30 16:53:35,515 DEBUG [RS:0;jenkins-hbase4:45193] zookeeper.ZKUtil(162): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:35,516 WARN [RS:0;jenkins-hbase4:45193] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-30 16:53:35,516 INFO [RS:0;jenkins-hbase4:45193] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-30 16:53:35,516 DEBUG [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(1947): logDir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/WALs/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:35,517 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,45193,1680195214285] 2023-03-30 16:53:35,525 DEBUG [RS:0;jenkins-hbase4:45193] zookeeper.ZKUtil(162): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:35,534 DEBUG [RS:0;jenkins-hbase4:45193] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-30 16:53:35,542 INFO [RS:0;jenkins-hbase4:45193] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-30 16:53:35,560 INFO [RS:0;jenkins-hbase4:45193] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-30 16:53:35,563 INFO [RS:0;jenkins-hbase4:45193] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-30 16:53:35,563 INFO [RS:0;jenkins-hbase4:45193] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-30 16:53:35,564 INFO [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-30 16:53:35,570 INFO [RS:0;jenkins-hbase4:45193] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-30 16:53:35,571 DEBUG [RS:0;jenkins-hbase4:45193] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-30 16:53:35,571 DEBUG [RS:0;jenkins-hbase4:45193] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-30 16:53:35,571 DEBUG [RS:0;jenkins-hbase4:45193] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-30 16:53:35,571 DEBUG [RS:0;jenkins-hbase4:45193] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-30 16:53:35,571 DEBUG [RS:0;jenkins-hbase4:45193] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-30 16:53:35,572 DEBUG [RS:0;jenkins-hbase4:45193] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-30 16:53:35,572 DEBUG [RS:0;jenkins-hbase4:45193] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-30 16:53:35,572 DEBUG [RS:0;jenkins-hbase4:45193] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-30 16:53:35,572 DEBUG [RS:0;jenkins-hbase4:45193] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-30 16:53:35,572 DEBUG [RS:0;jenkins-hbase4:45193] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-30 16:53:35,573 INFO [RS:0;jenkins-hbase4:45193] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-30 16:53:35,573 INFO [RS:0;jenkins-hbase4:45193] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-30 16:53:35,573 INFO [RS:0;jenkins-hbase4:45193] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-30 16:53:35,588 INFO [RS:0;jenkins-hbase4:45193] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-30 16:53:35,590 INFO [RS:0;jenkins-hbase4:45193] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45193,1680195214285-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-30 16:53:35,605 INFO [RS:0;jenkins-hbase4:45193] regionserver.Replication(203): jenkins-hbase4.apache.org,45193,1680195214285 started 2023-03-30 16:53:35,606 INFO [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,45193,1680195214285, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:45193, sessionid=0x10136583d700001 2023-03-30 16:53:35,606 DEBUG [RS:0;jenkins-hbase4:45193] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-30 16:53:35,606 DEBUG [RS:0;jenkins-hbase4:45193] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:35,606 DEBUG [RS:0;jenkins-hbase4:45193] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,45193,1680195214285' 2023-03-30 16:53:35,606 DEBUG [RS:0;jenkins-hbase4:45193] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:35,607 DEBUG [RS:0;jenkins-hbase4:45193] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:35,607 DEBUG [RS:0;jenkins-hbase4:45193] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-30 16:53:35,608 DEBUG [RS:0;jenkins-hbase4:45193] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-30 16:53:35,608 DEBUG [RS:0;jenkins-hbase4:45193] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:35,608 DEBUG [RS:0;jenkins-hbase4:45193] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,45193,1680195214285' 2023-03-30 16:53:35,608 DEBUG [RS:0;jenkins-hbase4:45193] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-30 16:53:35,608 DEBUG [RS:0;jenkins-hbase4:45193] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-30 16:53:35,609 DEBUG [RS:0;jenkins-hbase4:45193] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-30 16:53:35,609 INFO [RS:0;jenkins-hbase4:45193] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-30 16:53:35,609 INFO [RS:0;jenkins-hbase4:45193] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-30 16:53:35,720 INFO [RS:0;jenkins-hbase4:45193] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45193%2C1680195214285, suffix=, logDir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/WALs/jenkins-hbase4.apache.org,45193,1680195214285, archiveDir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/oldWALs, maxLogs=32 2023-03-30 16:53:35,738 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35955,DS-b8408660-09bd-410e-b9fa-5156fd0bd779,DISK] 2023-03-30 16:53:35,742 INFO [RS:0;jenkins-hbase4:45193] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/WALs/jenkins-hbase4.apache.org,45193,1680195214285/jenkins-hbase4.apache.org%2C45193%2C1680195214285.1680195215722 2023-03-30 16:53:35,742 DEBUG [RS:0;jenkins-hbase4:45193] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35955,DS-b8408660-09bd-410e-b9fa-5156fd0bd779,DISK]] 2023-03-30 16:53:35,778 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-30 16:53:35,780 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-30 16:53:35,780 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c 2023-03-30 16:53:36,196 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:53:36,199 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-30 16:53:36,203 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/info 2023-03-30 16:53:36,204 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-30 16:53:36,206 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:53:36,206 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-30 16:53:36,209 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/rep_barrier 2023-03-30 16:53:36,210 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-30 16:53:36,211 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:53:36,211 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-30 16:53:36,214 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/table 2023-03-30 16:53:36,215 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-30 16:53:36,216 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:53:36,217 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740 2023-03-30 16:53:36,219 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740 2023-03-30 16:53:36,223 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 16:53:36,226 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-30 16:53:36,233 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-30 16:53:36,237 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66420985, jitterRate=-0.010250195860862732}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 16:53:36,237 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-30 16:53:36,237 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-30 16:53:36,237 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-30 16:53:36,237 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-30 16:53:36,237 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-30 16:53:36,237 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-30 16:53:36,238 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-30 16:53:36,239 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-30 16:53:36,245 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-03-30 16:53:36,245 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-03-30 16:53:36,254 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-03-30 16:53:36,269 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-03-30 16:53:36,271 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-03-30 16:53:36,423 DEBUG [jenkins-hbase4:40565] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-03-30 16:53:36,426 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,45193,1680195214285, state=OPENING 2023-03-30 16:53:36,433 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-03-30 16:53:36,436 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-30 16:53:36,436 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-30 16:53:36,438 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,45193,1680195214285}] 2023-03-30 16:53:36,620 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:36,623 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-30 16:53:36,626 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41726, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-30 16:53:36,637 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-03-30 16:53:36,637 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-30 16:53:36,640 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45193%2C1680195214285.meta, suffix=.meta, logDir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/WALs/jenkins-hbase4.apache.org,45193,1680195214285, archiveDir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/oldWALs, maxLogs=32 2023-03-30 16:53:36,656 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35955,DS-b8408660-09bd-410e-b9fa-5156fd0bd779,DISK] 2023-03-30 16:53:36,660 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/WALs/jenkins-hbase4.apache.org,45193,1680195214285/jenkins-hbase4.apache.org%2C45193%2C1680195214285.meta.1680195216641.meta 2023-03-30 16:53:36,661 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35955,DS-b8408660-09bd-410e-b9fa-5156fd0bd779,DISK]] 2023-03-30 16:53:36,661 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-03-30 16:53:36,663 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-03-30 16:53:36,683 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-03-30 16:53:36,690 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-03-30 16:53:36,696 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-03-30 16:53:36,696 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:53:36,697 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-03-30 16:53:36,697 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-03-30 16:53:36,699 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-30 16:53:36,701 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/info 2023-03-30 16:53:36,701 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/info 2023-03-30 16:53:36,702 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-30 16:53:36,703 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:53:36,703 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-30 16:53:36,705 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/rep_barrier 2023-03-30 16:53:36,705 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/rep_barrier 2023-03-30 16:53:36,705 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-30 16:53:36,706 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:53:36,706 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-30 16:53:36,708 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/table 2023-03-30 16:53:36,708 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/table 2023-03-30 16:53:36,708 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-30 16:53:36,709 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:53:36,711 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740 2023-03-30 16:53:36,713 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740 2023-03-30 16:53:36,717 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 16:53:36,719 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-30 16:53:36,721 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58848189, jitterRate=-0.12309364974498749}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 16:53:36,721 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-30 16:53:36,730 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680195216613 2023-03-30 16:53:36,748 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-03-30 16:53:36,748 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-03-30 16:53:36,749 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,45193,1680195214285, state=OPEN 2023-03-30 16:53:36,752 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-03-30 16:53:36,752 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-30 16:53:36,756 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-03-30 16:53:36,756 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,45193,1680195214285 in 314 msec 2023-03-30 16:53:36,761 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-03-30 16:53:36,761 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 504 msec 2023-03-30 16:53:36,766 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.5240 sec 2023-03-30 16:53:36,766 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680195216766, completionTime=-1 2023-03-30 16:53:36,766 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-03-30 16:53:36,767 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-03-30 16:53:36,828 DEBUG [hconnection-0x4f91ae25-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:53:36,832 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41734, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:53:36,849 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-03-30 16:53:36,849 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680195276849 2023-03-30 16:53:36,850 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680195336850 2023-03-30 16:53:36,850 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 83 msec 2023-03-30 16:53:36,872 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40565,1680195213071-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-30 16:53:36,873 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40565,1680195213071-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-30 16:53:36,873 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40565,1680195213071-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-30 16:53:36,874 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:40565, period=300000, unit=MILLISECONDS is enabled. 2023-03-30 16:53:36,878 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-03-30 16:53:36,882 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-03-30 16:53:36,891 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-03-30 16:53:36,893 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-30 16:53:36,902 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-03-30 16:53:36,905 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-03-30 16:53:36,907 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-30 16:53:36,930 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/hbase/namespace/e324c540a5e4b217db9fa3051ef755ac 2023-03-30 16:53:36,932 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/hbase/namespace/e324c540a5e4b217db9fa3051ef755ac empty. 2023-03-30 16:53:36,933 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/hbase/namespace/e324c540a5e4b217db9fa3051ef755ac 2023-03-30 16:53:36,933 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-03-30 16:53:37,354 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-03-30 16:53:37,357 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => e324c540a5e4b217db9fa3051ef755ac, NAME => 'hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp 2023-03-30 16:53:37,369 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:53:37,370 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing e324c540a5e4b217db9fa3051ef755ac, disabling compactions & flushes 2023-03-30 16:53:37,370 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac. 2023-03-30 16:53:37,370 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac. 2023-03-30 16:53:37,370 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac. after waiting 0 ms 2023-03-30 16:53:37,370 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac. 2023-03-30 16:53:37,370 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac. 2023-03-30 16:53:37,370 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for e324c540a5e4b217db9fa3051ef755ac: 2023-03-30 16:53:37,374 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-03-30 16:53:37,390 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680195217378"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680195217378"}]},"ts":"1680195217378"} 2023-03-30 16:53:37,418 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-30 16:53:37,420 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-30 16:53:37,424 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195217420"}]},"ts":"1680195217420"} 2023-03-30 16:53:37,429 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-03-30 16:53:37,439 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e324c540a5e4b217db9fa3051ef755ac, ASSIGN}] 2023-03-30 16:53:37,441 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e324c540a5e4b217db9fa3051ef755ac, ASSIGN 2023-03-30 16:53:37,443 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=e324c540a5e4b217db9fa3051ef755ac, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,45193,1680195214285; forceNewPlan=false, retain=false 2023-03-30 16:53:37,594 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=e324c540a5e4b217db9fa3051ef755ac, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:37,595 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680195217594"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680195217594"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680195217594"}]},"ts":"1680195217594"} 2023-03-30 16:53:37,599 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure e324c540a5e4b217db9fa3051ef755ac, server=jenkins-hbase4.apache.org,45193,1680195214285}] 2023-03-30 16:53:37,760 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac. 2023-03-30 16:53:37,761 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e324c540a5e4b217db9fa3051ef755ac, NAME => 'hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac.', STARTKEY => '', ENDKEY => ''} 2023-03-30 16:53:37,762 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace e324c540a5e4b217db9fa3051ef755ac 2023-03-30 16:53:37,763 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:53:37,763 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e324c540a5e4b217db9fa3051ef755ac 2023-03-30 16:53:37,763 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e324c540a5e4b217db9fa3051ef755ac 2023-03-30 16:53:37,766 INFO [StoreOpener-e324c540a5e4b217db9fa3051ef755ac-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region e324c540a5e4b217db9fa3051ef755ac 2023-03-30 16:53:37,768 DEBUG [StoreOpener-e324c540a5e4b217db9fa3051ef755ac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/namespace/e324c540a5e4b217db9fa3051ef755ac/info 2023-03-30 16:53:37,768 DEBUG [StoreOpener-e324c540a5e4b217db9fa3051ef755ac-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/namespace/e324c540a5e4b217db9fa3051ef755ac/info 2023-03-30 16:53:37,769 INFO [StoreOpener-e324c540a5e4b217db9fa3051ef755ac-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e324c540a5e4b217db9fa3051ef755ac columnFamilyName info 2023-03-30 16:53:37,770 INFO [StoreOpener-e324c540a5e4b217db9fa3051ef755ac-1] regionserver.HStore(310): Store=e324c540a5e4b217db9fa3051ef755ac/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:53:37,772 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/namespace/e324c540a5e4b217db9fa3051ef755ac 2023-03-30 16:53:37,772 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/namespace/e324c540a5e4b217db9fa3051ef755ac 2023-03-30 16:53:37,777 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e324c540a5e4b217db9fa3051ef755ac 2023-03-30 16:53:37,780 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/namespace/e324c540a5e4b217db9fa3051ef755ac/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-30 16:53:37,781 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e324c540a5e4b217db9fa3051ef755ac; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71076124, jitterRate=0.05911678075790405}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-30 16:53:37,781 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e324c540a5e4b217db9fa3051ef755ac: 2023-03-30 16:53:37,783 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac., pid=6, masterSystemTime=1680195217753 2023-03-30 16:53:37,788 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac. 2023-03-30 16:53:37,788 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac. 2023-03-30 16:53:37,789 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=e324c540a5e4b217db9fa3051ef755ac, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:37,790 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680195217789"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680195217789"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680195217789"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680195217789"}]},"ts":"1680195217789"} 2023-03-30 16:53:37,797 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-03-30 16:53:37,798 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure e324c540a5e4b217db9fa3051ef755ac, server=jenkins-hbase4.apache.org,45193,1680195214285 in 195 msec 2023-03-30 16:53:37,801 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-03-30 16:53:37,802 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=e324c540a5e4b217db9fa3051ef755ac, ASSIGN in 358 msec 2023-03-30 16:53:37,803 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-30 16:53:37,803 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195217803"}]},"ts":"1680195217803"} 2023-03-30 16:53:37,806 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-03-30 16:53:37,811 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-03-30 16:53:37,811 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-03-30 16:53:37,812 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-03-30 16:53:37,812 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-30 16:53:37,819 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 917 msec 2023-03-30 16:53:37,855 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-03-30 16:53:37,874 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-30 16:53:37,880 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 38 msec 2023-03-30 16:53:37,888 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-03-30 16:53:37,900 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-30 16:53:37,904 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 15 msec 2023-03-30 16:53:37,914 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-03-30 16:53:37,918 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-03-30 16:53:37,918 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.554sec 2023-03-30 16:53:37,920 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-03-30 16:53:37,921 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-03-30 16:53:37,921 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-03-30 16:53:37,922 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40565,1680195213071-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-03-30 16:53:37,923 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40565,1680195213071-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-03-30 16:53:37,934 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-03-30 16:53:37,936 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x610f60f0 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:53:37,941 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@27bb171b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:53:37,954 DEBUG [hconnection-0x7f2af308-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:53:37,965 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41746, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:53:37,975 INFO [Listener at localhost/38075] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,40565,1680195213071 2023-03-30 16:53:37,995 INFO [Listener at localhost/38075] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=307, ProcessCount=169, AvailableMemoryMB=5471 2023-03-30 16:53:38,011 DEBUG [Listener at localhost/38075] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-30 16:53:38,015 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41304, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-30 16:53:38,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-30 16:53:38,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-30 16:53:38,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-30 16:53:38,031 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-30 16:53:38,034 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-30 16:53:38,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-03-30 16:53:38,037 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:38,038 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d empty. 2023-03-30 16:53:38,040 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:38,040 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 16:53:38,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-30 16:53:38,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-30 16:53:38,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-30 16:53:38,459 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-30 16:53:38,461 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4600ee2f6bff5cc4f7b8bf671d05894d, NAME => 'TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp 2023-03-30 16:53:38,473 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:53:38,473 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 4600ee2f6bff5cc4f7b8bf671d05894d, disabling compactions & flushes 2023-03-30 16:53:38,473 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:38,473 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:38,473 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. after waiting 0 ms 2023-03-30 16:53:38,473 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:38,473 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:38,473 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:38,477 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-30 16:53:38,479 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195218479"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680195218479"}]},"ts":"1680195218479"} 2023-03-30 16:53:38,482 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-30 16:53:38,483 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-30 16:53:38,484 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195218483"}]},"ts":"1680195218483"} 2023-03-30 16:53:38,486 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-30 16:53:38,491 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4600ee2f6bff5cc4f7b8bf671d05894d, ASSIGN}] 2023-03-30 16:53:38,493 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4600ee2f6bff5cc4f7b8bf671d05894d, ASSIGN 2023-03-30 16:53:38,495 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=4600ee2f6bff5cc4f7b8bf671d05894d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,45193,1680195214285; forceNewPlan=false, retain=false 2023-03-30 16:53:38,646 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=4600ee2f6bff5cc4f7b8bf671d05894d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:38,646 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195218646"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680195218646"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680195218646"}]},"ts":"1680195218646"} 2023-03-30 16:53:38,650 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285}] 2023-03-30 16:53:38,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-30 16:53:38,809 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:38,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4600ee2f6bff5cc4f7b8bf671d05894d, NAME => 'TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.', STARTKEY => '', ENDKEY => ''} 2023-03-30 16:53:38,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:38,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:53:38,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:38,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:38,812 INFO [StoreOpener-4600ee2f6bff5cc4f7b8bf671d05894d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:38,814 DEBUG [StoreOpener-4600ee2f6bff5cc4f7b8bf671d05894d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A 2023-03-30 16:53:38,814 DEBUG [StoreOpener-4600ee2f6bff5cc4f7b8bf671d05894d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A 2023-03-30 16:53:38,816 INFO [StoreOpener-4600ee2f6bff5cc4f7b8bf671d05894d-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:53:38,816 INFO [StoreOpener-4600ee2f6bff5cc4f7b8bf671d05894d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4600ee2f6bff5cc4f7b8bf671d05894d columnFamilyName A 2023-03-30 16:53:38,817 INFO [StoreOpener-4600ee2f6bff5cc4f7b8bf671d05894d-1] regionserver.HStore(310): Store=4600ee2f6bff5cc4f7b8bf671d05894d/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:53:38,817 INFO [StoreOpener-4600ee2f6bff5cc4f7b8bf671d05894d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:38,819 DEBUG [StoreOpener-4600ee2f6bff5cc4f7b8bf671d05894d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B 2023-03-30 16:53:38,819 DEBUG [StoreOpener-4600ee2f6bff5cc4f7b8bf671d05894d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B 2023-03-30 16:53:38,820 INFO [StoreOpener-4600ee2f6bff5cc4f7b8bf671d05894d-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:53:38,820 INFO [StoreOpener-4600ee2f6bff5cc4f7b8bf671d05894d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4600ee2f6bff5cc4f7b8bf671d05894d columnFamilyName B 2023-03-30 16:53:38,821 INFO [StoreOpener-4600ee2f6bff5cc4f7b8bf671d05894d-1] regionserver.HStore(310): Store=4600ee2f6bff5cc4f7b8bf671d05894d/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:53:38,821 INFO [StoreOpener-4600ee2f6bff5cc4f7b8bf671d05894d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:38,823 DEBUG [StoreOpener-4600ee2f6bff5cc4f7b8bf671d05894d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C 2023-03-30 16:53:38,823 DEBUG [StoreOpener-4600ee2f6bff5cc4f7b8bf671d05894d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C 2023-03-30 16:53:38,823 INFO [StoreOpener-4600ee2f6bff5cc4f7b8bf671d05894d-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:53:38,823 INFO [StoreOpener-4600ee2f6bff5cc4f7b8bf671d05894d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4600ee2f6bff5cc4f7b8bf671d05894d columnFamilyName C 2023-03-30 16:53:38,824 INFO [StoreOpener-4600ee2f6bff5cc4f7b8bf671d05894d-1] regionserver.HStore(310): Store=4600ee2f6bff5cc4f7b8bf671d05894d/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:53:38,825 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:38,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:38,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:38,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 16:53:38,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:38,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-30 16:53:38,836 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4600ee2f6bff5cc4f7b8bf671d05894d; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60160358, jitterRate=-0.1035408079624176}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 16:53:38,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:38,838 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., pid=11, masterSystemTime=1680195218803 2023-03-30 16:53:38,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:38,840 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:38,841 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=4600ee2f6bff5cc4f7b8bf671d05894d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:38,842 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195218841"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680195218841"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680195218841"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680195218841"}]},"ts":"1680195218841"} 2023-03-30 16:53:38,848 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-03-30 16:53:38,848 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 in 195 msec 2023-03-30 16:53:38,851 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-03-30 16:53:38,852 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4600ee2f6bff5cc4f7b8bf671d05894d, ASSIGN in 357 msec 2023-03-30 16:53:38,853 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-30 16:53:38,853 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195218853"}]},"ts":"1680195218853"} 2023-03-30 16:53:38,855 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-30 16:53:38,859 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-30 16:53:38,861 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 831 msec 2023-03-30 16:53:39,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-30 16:53:39,160 INFO [Listener at localhost/38075] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-03-30 16:53:39,164 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x0013510a to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:53:39,170 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a2f833f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:53:39,172 DEBUG [Listener at localhost/38075] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:53:39,176 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41756, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:53:39,179 DEBUG [Listener at localhost/38075] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-30 16:53:39,181 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41312, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-30 16:53:39,189 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x12adc6a1 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:53:39,194 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@75f15925, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:53:39,194 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x2732b8c5 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:53:39,234 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2411a862, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:53:39,234 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x72ea4a65 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:53:39,243 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e82daee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:53:39,244 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x1645b6b0 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:53:39,260 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e8196e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:53:39,261 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x7b52aa42 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:53:39,276 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2e2005d0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:53:39,277 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x6ec07124 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:53:39,291 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bc6d729, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:53:39,292 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x51ef1151 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:53:39,320 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c4b1319, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:53:39,320 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x00375c9d to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:53:39,325 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@64804459, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:53:39,326 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x2ec2fae8 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:53:39,331 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@142b9819, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:53:39,338 DEBUG [hconnection-0x9312249-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:53:39,338 DEBUG [hconnection-0x6fed7337-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:53:39,340 DEBUG [hconnection-0x4c2fffd1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:53:39,340 DEBUG [hconnection-0x372d4e28-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:53:39,345 DEBUG [hconnection-0x74b555f4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:53:39,347 DEBUG [hconnection-0x637b85c0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:53:39,348 DEBUG [hconnection-0x66595227-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:53:39,354 DEBUG [hconnection-0x4d06a665-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:53:39,359 DEBUG [hconnection-0x524c453b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:53:39,360 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41770, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:53:39,361 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41798, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:53:39,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:39,368 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41836, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:53:39,383 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41762, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:53:39,389 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41792, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:53:39,389 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41828, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:53:39,396 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41778, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:53:39,397 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41814, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:53:39,398 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41852, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:53:39,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:39,413 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:39,413 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:39,415 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:39,415 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:39,416 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,416 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:39,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:39,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:39,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:39,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:39,421 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:39,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:39,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:39,426 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,426 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:39,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:39,437 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:39,437 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:39,441 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:39,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:39,453 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:39,453 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:39,460 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:39,460 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 16:53:39,460 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:39,453 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:39,461 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:39,463 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:39,463 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:39,463 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,463 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:39,463 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:39,466 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:39,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:39,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:39,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:39,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:39,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:39,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:39,479 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73222e4e[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:39,479 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:39,480 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,480 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,480 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:39,482 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:39,482 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,482 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:39,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,483 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:39,483 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:39,483 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:39,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:39,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:39,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:39,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:39,483 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,484 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:39,487 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:39,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:39,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:39,488 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:39,488 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:39,488 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:39,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:39,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:39,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:39,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:39,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:39,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:39,494 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:39,494 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:39,495 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:39,495 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:39,495 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:39,505 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,505 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:39,505 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:39,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:39,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:39,509 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:39,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:39,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:39,511 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:39,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,524 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:39,524 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:39,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:39,524 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:39,524 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:39,524 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,524 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,525 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,525 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:39,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:39,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:39,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:39,524 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:39,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:39,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:39,538 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:39,538 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:39,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:39,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195279606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:39,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195279607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:39,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195279609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:39,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195279613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:39,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195279619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,656 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:39,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:39,680 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/29b27f783da2410890836b3acf842c73 2023-03-30 16:53:39,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:39,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:39,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:39,720 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:39,720 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:39,721 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:39,721 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:39,722 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,722 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:39,724 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:39,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:39,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:39,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:39,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,726 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,727 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,727 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:39,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:39,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:39,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:39,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:39,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:39,737 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:39,737 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:39,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:39,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195279734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,739 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:39,740 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:39,740 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:39,741 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:39,741 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:39,741 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:39,741 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:39,742 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:39,742 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:39,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:39,742 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,742 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:39,742 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:39,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195279737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:39,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195279738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:39,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195279738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:39,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195279738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,746 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,746 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:39,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:39,747 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,747 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:39,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:39,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:39,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:39,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:39,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:39,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73c052dc[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:39,753 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:39,753 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,755 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,755 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:39,755 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:39,755 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:39,755 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,755 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:39,758 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:39,758 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:39,758 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:39,758 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:39,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:39,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:39,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:39,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:39,765 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:39,765 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:39,765 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:39,765 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:39,765 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:39,771 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,771 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:39,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:39,771 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:39,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:39,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:39,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:39,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:39,773 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:39,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:39,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,775 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:39,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:39,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,786 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:39,786 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:39,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:39,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:39,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:39,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:39,793 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,793 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:39,793 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:39,793 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,793 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,793 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,793 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,795 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:39,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:39,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:39,798 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:39,798 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:39,834 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/82c8c3a2310548479c242a44396d5704 2023-03-30 16:53:39,898 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:39,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:39,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:39,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:39,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:39,924 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:39,925 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:39,925 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:39,925 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:39,926 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,926 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:39,929 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:39,929 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:39,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:39,929 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,929 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:39,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:39,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:39,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:39,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:39,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:39,931 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:39,931 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:39,933 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:39,933 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:39,934 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:39,934 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:39,934 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:39,934 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:39,934 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:39,934 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:39,935 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:39,935 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,935 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:39,935 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:39,941 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:39,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:39,942 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:39,943 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,943 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:39,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:39,944 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:39,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:39,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195279943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:39,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:39,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@608e900[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:39,947 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:39,947 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:39,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195279946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,950 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,950 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:39,950 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:39,950 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:39,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:39,950 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195279948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,950 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:39,951 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:39,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195279949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:39,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195279949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,955 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,955 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:39,955 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:39,955 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:39,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:39,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:39,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:39,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:39,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:39,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:39,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:39,963 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:39,963 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:39,963 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:39,963 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:39,963 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:39,967 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,967 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:39,967 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:39,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:39,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:39,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:39,968 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:39,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:39,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:39,972 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:39,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,986 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:39,986 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:39,986 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,986 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:39,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:39,986 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:39,991 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:39,991 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:39,991 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,991 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:39,991 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:39,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:39,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:39,992 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:39,992 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:39,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:39,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:39,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:40,092 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:40,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:40,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:40,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:40,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:40,105 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:40,105 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:40,106 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:40,106 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:40,107 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:40,107 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:40,110 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,110 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,111 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:40,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:40,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:40,111 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:40,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:40,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:40,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:40,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:40,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:40,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:40,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:40,115 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:40,116 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:40,117 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:40,117 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:40,117 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:40,117 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:40,117 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:40,117 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:40,117 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:40,118 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:40,118 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:40,118 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,118 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:40,118 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:40,122 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,122 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:40,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:40,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:40,123 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:40,123 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:40,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:40,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:40,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:40,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:40,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:40,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27d67fb1[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:40,126 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:40,126 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:40,127 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:40,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:40,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:40,127 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:40,127 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:40,127 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:40,128 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,128 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:40,133 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,133 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:40,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:40,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:40,133 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:40,133 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:40,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:40,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:40,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:40,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:40,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:40,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:40,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:40,139 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:40,139 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:40,139 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:40,139 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:40,139 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:40,146 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:40,146 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:40,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:40,146 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:40,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:40,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:40,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:40,149 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:40,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:40,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:40,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,151 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:40,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:40,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,172 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,172 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:40,172 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:40,172 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:40,172 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:40,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:40,172 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:40,173 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:40,173 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:40,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:40,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:40,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:40,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:40,173 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:40,173 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:40,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:40,173 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:40,180 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:40,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:40,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195280252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:40,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195280256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:40,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195280256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:40,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195280261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:40,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195280249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,280 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:40,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:40,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:40,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:40,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:40,312 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:40,312 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:40,317 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:40,317 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:40,320 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:40,320 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:40,322 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,322 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:40,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:40,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:40,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:40,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:40,324 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/a31fbccbf633462ab419b9448c8380df 2023-03-30 16:53:40,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:40,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:40,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:40,324 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,325 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:40,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:40,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:40,328 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:40,328 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:40,332 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:40,334 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:40,334 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:40,334 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:40,334 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:40,334 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:40,334 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:40,341 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:40,341 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:40,341 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,341 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:40,341 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:40,344 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,344 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:40,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:40,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:40,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:40,349 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/29b27f783da2410890836b3acf842c73 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/29b27f783da2410890836b3acf842c73 2023-03-30 16:53:40,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:40,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45e4755b[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:40,349 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:40,349 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:40,350 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:40,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:40,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:40,351 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:40,351 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:40,351 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:40,353 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:40,353 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:40,353 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:40,353 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,353 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:40,354 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:40,354 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:40,354 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:40,354 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:40,354 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:40,354 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:40,370 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:40,370 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:40,370 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:40,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:40,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:40,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:40,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:40,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:40,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:40,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:40,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:40,371 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:40,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:40,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:40,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:40,373 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:40,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:40,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,381 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/29b27f783da2410890836b3acf842c73, entries=150, sequenceid=14, filesize=11.6 K 2023-03-30 16:53:40,381 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,381 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:40,381 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:40,381 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:40,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:40,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:40,382 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:40,382 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,382 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:40,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:40,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:40,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:40,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:40,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:40,382 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:40,382 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:40,383 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:40,383 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:40,383 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:40,395 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/82c8c3a2310548479c242a44396d5704 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/82c8c3a2310548479c242a44396d5704 2023-03-30 16:53:40,417 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/82c8c3a2310548479c242a44396d5704, entries=150, sequenceid=14, filesize=11.6 K 2023-03-30 16:53:40,420 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/a31fbccbf633462ab419b9448c8380df as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/a31fbccbf633462ab419b9448c8380df 2023-03-30 16:53:40,446 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/a31fbccbf633462ab419b9448c8380df, entries=150, sequenceid=14, filesize=11.6 K 2023-03-30 16:53:40,450 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 4600ee2f6bff5cc4f7b8bf671d05894d in 990ms, sequenceid=14, compaction requested=false 2023-03-30 16:53:40,460 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-30 16:53:40,463 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:40,483 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:40,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:40,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:40,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:40,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:40,515 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:40,516 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:40,526 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:40,526 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:40,531 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:40,531 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:40,533 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:40,533 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:40,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:40,534 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,534 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:40,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:40,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:40,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:40,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:40,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:40,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:40,537 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:40,537 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:40,537 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:40,538 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:40,538 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:40,538 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:40,538 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:40,538 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:40,539 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-30 16:53:40,539 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:40,539 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:40,540 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:40,540 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:40,540 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:40,540 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:40,647 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/f31b411b148a4f7fb23bab8977c8e89a 2023-03-30 16:53:40,742 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/a6701ff30abc435280e99068521b43b6 2023-03-30 16:53:40,780 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:40,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:40,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:40,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195280800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:40,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195280801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:40,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195280801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:40,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195280822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:40,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195280822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:40,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195280919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:40,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195280919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:40,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195280920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:40,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195280932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:40,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:40,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195280938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:41,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195281123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:41,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195281125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:41,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195281125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:41,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195281138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:41,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195281141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,263 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/4d1ec7888d0c40809b256b0bb0ad6e8a 2023-03-30 16:53:41,275 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/f31b411b148a4f7fb23bab8977c8e89a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/f31b411b148a4f7fb23bab8977c8e89a 2023-03-30 16:53:41,285 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/f31b411b148a4f7fb23bab8977c8e89a, entries=150, sequenceid=33, filesize=11.6 K 2023-03-30 16:53:41,288 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/a6701ff30abc435280e99068521b43b6 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/a6701ff30abc435280e99068521b43b6 2023-03-30 16:53:41,305 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/a6701ff30abc435280e99068521b43b6, entries=150, sequenceid=33, filesize=11.6 K 2023-03-30 16:53:41,308 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/4d1ec7888d0c40809b256b0bb0ad6e8a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/4d1ec7888d0c40809b256b0bb0ad6e8a 2023-03-30 16:53:41,326 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/4d1ec7888d0c40809b256b0bb0ad6e8a, entries=150, sequenceid=33, filesize=11.6 K 2023-03-30 16:53:41,336 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 4600ee2f6bff5cc4f7b8bf671d05894d in 796ms, sequenceid=33, compaction requested=false 2023-03-30 16:53:41,337 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:41,337 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:41,337 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:41,338 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:41,338 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,338 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:41,338 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:41,342 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,342 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:41,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:41,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:41,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:41,345 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,345 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:41,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:41,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:41,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34439989[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:41,348 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:41,348 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,351 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,351 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:41,351 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:41,351 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:41,352 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,352 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:41,356 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:41,356 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:41,356 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:41,358 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:41,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:41,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:41,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:41,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:41,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:41,365 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:41,365 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:41,365 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:41,366 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:41,366 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:41,372 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:41,373 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,373 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:41,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:41,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:41,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:41,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,376 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:41,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:41,383 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,387 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,388 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:41,388 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:41,388 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:41,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:41,388 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:41,388 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,388 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,388 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,392 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:41,392 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:41,392 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:41,392 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:41,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:41,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:41,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:41,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:41,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:41,434 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-30 16:53:41,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:41,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:41,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:41,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:41,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:41,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:41,492 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:41,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:41,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:41,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195281475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:41,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195281486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:41,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195281486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:41,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:41,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195281492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:41,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195281499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:41,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:41,506 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:41,506 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:41,507 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:41,507 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:41,508 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,509 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:41,511 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:41,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:41,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:41,512 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:41,513 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,514 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:41,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:41,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:41,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:41,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:41,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:41,515 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:41,515 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:41,516 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:41,519 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:41,519 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:41,519 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:41,519 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:41,519 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:41,519 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:41,520 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:41,520 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:41,520 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,520 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:41,520 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:41,524 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,524 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:41,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:41,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:41,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:41,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:41,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:41,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:41,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@754226e8[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:41,528 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:41,528 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,529 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,529 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:41,529 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:41,530 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:41,530 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,530 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:41,532 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:41,532 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:41,532 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:41,532 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:41,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:41,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:41,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:41,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:41,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:41,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:41,537 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:41,537 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:41,537 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:41,537 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:41,537 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:41,539 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-03-30 16:53:41,545 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-03-30 16:53:41,551 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,551 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:41,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:41,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,551 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:41,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:41,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:41,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:41,554 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:41,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,560 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,565 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:41,565 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:41,565 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:41,565 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:41,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:41,565 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:41,565 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,565 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,565 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,565 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:41,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:41,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:41,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:41,566 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:41,566 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:41,569 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:41,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:41,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195281600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:41,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:41,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195281601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195281603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:41,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195281604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:41,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195281606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,669 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:41,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:41,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:41,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:41,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:41,677 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:41,677 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:41,678 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:41,678 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:41,679 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,679 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:41,681 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:41,681 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:41,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:41,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:41,681 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,681 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:41,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:41,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:41,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:41,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:41,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:41,683 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:41,683 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:41,684 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:41,692 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:41,692 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:41,692 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:41,692 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:41,692 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:41,692 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:41,693 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:41,694 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:41,694 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,694 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:41,694 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:41,697 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,697 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:41,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:41,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:41,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:41,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,699 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,699 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:41,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:41,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:41,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61fde406[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:41,700 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:41,700 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,702 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,703 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:41,703 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:41,703 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:41,703 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,703 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:41,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:41,705 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:41,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:41,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:41,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:41,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:41,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:41,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:41,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:41,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:41,709 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:41,710 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:41,710 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:41,710 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:41,710 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:41,713 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:41,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:41,714 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,714 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:41,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:41,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:41,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:41,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,725 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:41,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,737 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:41,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,744 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:41,744 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,744 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:41,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:41,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:41,744 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:41,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:41,744 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:41,746 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:41,746 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,746 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,746 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,746 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,746 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:41,746 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:41,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:41,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:41,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:41,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:41,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195281804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:41,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195281809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:41,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195281809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:41,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195281811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:41,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195281812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,846 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:41,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:41,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:41,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:41,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:41,852 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:41,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:41,853 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:41,853 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:41,854 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,854 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:41,860 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:41,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:41,861 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:41,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:41,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:41,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:41,863 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,863 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:41,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:41,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:41,863 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:41,863 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:41,867 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:41,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:41,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:41,868 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:41,868 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:41,868 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:41,868 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:41,877 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:41,878 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:41,878 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,878 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:41,878 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:41,881 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,881 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:41,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:41,883 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,883 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:41,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:41,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:41,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d50bcf[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:41,891 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:41,891 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,896 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,897 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:41,897 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:41,897 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/91bbbca69cd74145ace260509072a55f 2023-03-30 16:53:41,897 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,897 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:41,897 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:41,901 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,901 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:41,901 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:41,901 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:41,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:41,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:41,911 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:41,911 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:41,911 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:41,911 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:41,911 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:41,916 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,916 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:41,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,917 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:41,921 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:41,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,923 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:41,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:41,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,930 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:41,930 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:41,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:41,930 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,931 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:41,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:41,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:41,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:41,931 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:41,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:41,931 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:41,931 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:41,931 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:41,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:41,931 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:41,931 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,932 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:41,932 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:41,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:42,031 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:42,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:42,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:42,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:42,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:42,037 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:42,037 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:42,038 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:42,038 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:42,039 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,039 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,040 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:42,041 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:42,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:42,041 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,041 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:42,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:42,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:42,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:42,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:42,042 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:42,043 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:42,045 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:42,045 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:42,046 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:42,046 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:42,046 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:42,046 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:42,046 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:42,047 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:42,047 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:42,047 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,047 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:42,047 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:42,049 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,049 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:42,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:42,050 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,050 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:42,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:42,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:42,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:42,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:42,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ff2556d[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:42,052 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:42,052 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,054 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,054 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:42,054 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:42,054 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:42,054 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,054 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:42,056 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:42,056 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,056 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:42,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:42,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:42,056 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:42,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:42,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:42,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:42,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:42,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:42,063 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:42,063 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:42,063 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:42,063 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:42,063 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:42,067 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,067 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:42,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:42,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:42,067 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:42,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:42,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,068 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:42,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:42,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,070 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:42,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,075 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:42,075 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,076 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:42,076 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,076 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:42,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:42,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:42,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:42,075 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:42,076 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,076 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,076 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,076 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,076 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:42,076 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:42,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:42,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:42,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:42,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195282108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:42,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195282115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:42,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195282116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:42,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195282117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:42,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195282121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,176 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:42,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:42,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:42,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:42,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:42,183 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:42,183 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:42,184 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:42,184 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:42,185 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,185 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,187 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,187 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:42,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:42,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:42,187 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,187 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:42,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:42,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:42,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:42,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:42,190 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:42,190 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:42,191 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:42,191 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:42,191 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:42,191 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:42,191 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:42,191 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:42,191 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:42,192 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:42,193 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:42,193 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,193 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:42,193 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:42,194 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,194 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:42,195 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:42,195 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:42,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:42,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:42,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b63531d[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:42,197 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:42,197 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,198 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,199 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,199 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:42,199 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:42,199 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:42,199 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:42,201 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,201 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:42,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:42,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:42,201 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:42,202 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:42,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:42,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:42,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:42,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:42,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:42,208 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:42,208 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:42,208 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:42,208 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:42,208 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:42,210 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,211 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:42,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,211 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:42,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:42,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:42,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:42,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:42,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,212 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:42,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,215 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:42,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,218 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:42,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:42,218 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:42,218 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:42,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:42,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:42,218 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:42,218 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,220 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,220 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,220 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,220 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,220 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:42,220 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:42,220 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:42,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:42,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:42,266 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-30 16:53:42,321 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:42,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:42,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:42,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:42,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:42,329 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:42,329 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:42,332 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:42,332 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:42,333 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,333 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,335 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:42,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:42,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:42,335 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,336 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,336 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:42,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:42,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:42,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:42,337 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:42,337 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:42,338 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:42,339 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:42,339 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:42,339 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:42,339 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:42,339 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:42,339 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:42,340 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:42,341 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:42,341 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,341 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:42,341 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:42,353 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,353 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,353 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:42,359 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:42,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:42,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:42,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:42,362 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/fcb762bb03404bbc9c16c5646ba66d97 2023-03-30 16:53:42,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:42,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:42,388 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:42,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@453053a3[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:42,388 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,392 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,393 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:42,393 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:42,393 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:42,394 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,394 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:42,397 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:42,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:42,397 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:42,398 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:42,398 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:42,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:42,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:42,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:42,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:42,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:42,408 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:42,408 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:42,408 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:42,408 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:42,408 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:42,410 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,410 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:42,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:42,411 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:42,412 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:42,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,415 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:42,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:42,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:42,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:42,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:42,420 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:42,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,420 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:42,420 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:42,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:42,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:42,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:42,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:42,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:42,521 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:42,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:42,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:42,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:42,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:42,535 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:42,535 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:42,537 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:42,537 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:42,537 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,537 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,541 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,541 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:42,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:42,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:42,542 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,542 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:42,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:42,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:42,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:42,552 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:42,552 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:42,553 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:42,565 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:42,565 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:42,566 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:42,566 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:42,566 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:42,566 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:42,566 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:42,566 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:42,566 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,566 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:42,566 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:42,568 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,568 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,569 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,569 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:42,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:42,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:42,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:42,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:42,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:42,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a14b3b9[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:42,577 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:42,577 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,581 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,581 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:42,581 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:42,581 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:42,583 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:42,583 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,583 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:42,583 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:42,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:42,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:42,584 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,584 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:42,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:42,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:42,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:42,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:42,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:42,587 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:42,587 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:42,587 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:42,588 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:42,588 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:42,590 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,590 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:42,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:42,590 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:42,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,593 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:42,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:42,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:42,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:42,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,596 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:42,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,600 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:42,600 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:42,600 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:42,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:42,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:42,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:42,600 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,601 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,600 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:42,601 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:42,601 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:42,601 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,602 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,602 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,602 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:42,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:42,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:42,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:42,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195282614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:42,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:42,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195282619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195282619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:42,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195282621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:42,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195282627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,702 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:42,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:42,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:42,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:42,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:42,732 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:42,732 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:42,733 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:42,733 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:42,734 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,734 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,737 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:42,737 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:42,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:42,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,738 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,738 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:42,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:42,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:42,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:42,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:42,739 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:42,739 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:42,740 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:42,747 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:42,748 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:42,748 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:42,748 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:42,748 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:42,748 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:42,748 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:42,748 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:42,748 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,748 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:42,748 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:42,751 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,751 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:42,752 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,752 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:42,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:42,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:42,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5770daab[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:42,756 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:42,756 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,758 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,758 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:42,758 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:42,758 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:42,760 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:42,760 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:42,760 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:42,765 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,766 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:42,766 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:42,766 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:42,766 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:42,766 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:42,766 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:42,766 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:42,768 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,768 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:42,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:42,769 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:42,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:42,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:42,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:42,770 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:42,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,772 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:42,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:42,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,776 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:42,776 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:42,776 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,777 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,777 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,777 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:42,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:42,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:42,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:42,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:42,777 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:42,778 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,778 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:42,778 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:42,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:42,783 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:42,784 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:42,809 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/98a966eb1c0a4488a9ebdb34df590fff 2023-03-30 16:53:42,825 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/91bbbca69cd74145ace260509072a55f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/91bbbca69cd74145ace260509072a55f 2023-03-30 16:53:42,842 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/91bbbca69cd74145ace260509072a55f, entries=150, sequenceid=49, filesize=11.6 K 2023-03-30 16:53:42,848 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/fcb762bb03404bbc9c16c5646ba66d97 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/fcb762bb03404bbc9c16c5646ba66d97 2023-03-30 16:53:42,862 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/fcb762bb03404bbc9c16c5646ba66d97, entries=150, sequenceid=49, filesize=11.6 K 2023-03-30 16:53:42,865 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/98a966eb1c0a4488a9ebdb34df590fff as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/98a966eb1c0a4488a9ebdb34df590fff 2023-03-30 16:53:42,874 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/98a966eb1c0a4488a9ebdb34df590fff, entries=150, sequenceid=49, filesize=11.6 K 2023-03-30 16:53:42,879 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 4600ee2f6bff5cc4f7b8bf671d05894d in 1445ms, sequenceid=49, compaction requested=true 2023-03-30 16:53:42,879 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:42,884 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:42,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:42,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:42,892 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:53:42,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:42,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:42,892 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:53:42,892 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:42,892 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:53:42,892 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:53:42,892 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:53:42,892 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:42,893 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:42,893 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:42,894 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,894 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,896 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:42,896 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:42,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:42,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,897 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:42,897 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:42,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:42,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:42,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:42,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:42,899 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:53:42,899 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:53:42,902 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/B is initiating minor compaction (all files) 2023-03-30 16:53:42,902 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/A is initiating minor compaction (all files) 2023-03-30 16:53:42,902 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/B in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:42,902 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/A in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:42,902 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/29b27f783da2410890836b3acf842c73, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/f31b411b148a4f7fb23bab8977c8e89a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/91bbbca69cd74145ace260509072a55f] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=34.8 K 2023-03-30 16:53:42,906 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 29b27f783da2410890836b3acf842c73, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680195219398 2023-03-30 16:53:42,909 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting f31b411b148a4f7fb23bab8977c8e89a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680195219597 2023-03-30 16:53:42,910 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 91bbbca69cd74145ace260509072a55f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680195220794 2023-03-30 16:53:42,902 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/82c8c3a2310548479c242a44396d5704, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/a6701ff30abc435280e99068521b43b6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/fcb762bb03404bbc9c16c5646ba66d97] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=34.8 K 2023-03-30 16:53:42,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:42,913 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:42,913 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:42,915 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:42,915 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:42,916 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:42,916 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:42,916 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:42,916 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:42,916 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 82c8c3a2310548479c242a44396d5704, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680195219398 2023-03-30 16:53:42,917 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-30 16:53:42,918 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:42,918 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:42,918 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:42,918 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:42,918 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:42,918 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:42,921 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting a6701ff30abc435280e99068521b43b6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680195219597 2023-03-30 16:53:42,922 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting fcb762bb03404bbc9c16c5646ba66d97, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680195220794 2023-03-30 16:53:42,999 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#B#compaction#11 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:53:42,999 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#A#compaction#10 average throughput is 0.34 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:53:43,240 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-30 16:53:43,241 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-03-30 16:53:43,424 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/5d9e9067be924258b1f917729735c8ff 2023-03-30 16:53:43,476 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/2fe13d19e14e49c49f5e1505655faa14 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/2fe13d19e14e49c49f5e1505655faa14 2023-03-30 16:53:43,481 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/ba389fa14dde4064a070b904b13a7509 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/ba389fa14dde4064a070b904b13a7509 2023-03-30 16:53:43,503 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/B of 4600ee2f6bff5cc4f7b8bf671d05894d into ba389fa14dde4064a070b904b13a7509(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:53:43,503 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/A of 4600ee2f6bff5cc4f7b8bf671d05894d into 2fe13d19e14e49c49f5e1505655faa14(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:53:43,503 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:43,504 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:43,504 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/B, priority=13, startTime=1680195222892; duration=0sec 2023-03-30 16:53:43,504 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/A, priority=13, startTime=1680195222883; duration=0sec 2023-03-30 16:53:43,506 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:53:43,506 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:53:43,506 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:53:43,510 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:53:43,510 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/C is initiating minor compaction (all files) 2023-03-30 16:53:43,510 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/C in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:43,511 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/a31fbccbf633462ab419b9448c8380df, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/4d1ec7888d0c40809b256b0bb0ad6e8a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/98a966eb1c0a4488a9ebdb34df590fff] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=34.8 K 2023-03-30 16:53:43,511 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting a31fbccbf633462ab419b9448c8380df, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680195219398 2023-03-30 16:53:43,512 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 4d1ec7888d0c40809b256b0bb0ad6e8a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680195219597 2023-03-30 16:53:43,513 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 98a966eb1c0a4488a9ebdb34df590fff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680195220794 2023-03-30 16:53:43,537 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#C#compaction#13 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:53:43,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:43,630 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:43,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:43,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195283649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:43,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:43,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195283649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:43,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:43,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195283651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:43,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:43,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195283654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:43,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:43,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195283654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:43,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:43,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195283755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:43,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:43,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195283758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:43,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:43,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195283758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:43,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:43,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195283763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:43,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:43,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195283764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:43,871 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/acecd5442a3c4e5dba63a830513000a5 2023-03-30 16:53:43,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:43,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195283959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:43,962 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:43,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195283961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:43,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:43,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195283962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:43,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:43,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195283970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:43,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:43,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195283971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:43,985 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/b484898c087d4f14bc44df334024faf5 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/b484898c087d4f14bc44df334024faf5 2023-03-30 16:53:43,996 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/C of 4600ee2f6bff5cc4f7b8bf671d05894d into b484898c087d4f14bc44df334024faf5(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:53:43,996 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:43,996 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/C, priority=13, startTime=1680195222892; duration=0sec 2023-03-30 16:53:43,996 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:53:44,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:44,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195284266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:44,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195284266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:44,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195284267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:44,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195284276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:44,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195284276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,300 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/26d909c968cb41958c7a053c0e442382 2023-03-30 16:53:44,311 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/5d9e9067be924258b1f917729735c8ff as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/5d9e9067be924258b1f917729735c8ff 2023-03-30 16:53:44,320 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/5d9e9067be924258b1f917729735c8ff, entries=150, sequenceid=65, filesize=11.6 K 2023-03-30 16:53:44,323 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/acecd5442a3c4e5dba63a830513000a5 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/acecd5442a3c4e5dba63a830513000a5 2023-03-30 16:53:44,338 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/acecd5442a3c4e5dba63a830513000a5, entries=150, sequenceid=65, filesize=11.6 K 2023-03-30 16:53:44,349 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/26d909c968cb41958c7a053c0e442382 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/26d909c968cb41958c7a053c0e442382 2023-03-30 16:53:44,367 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/26d909c968cb41958c7a053c0e442382, entries=150, sequenceid=65, filesize=11.6 K 2023-03-30 16:53:44,370 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 4600ee2f6bff5cc4f7b8bf671d05894d in 1453ms, sequenceid=65, compaction requested=false 2023-03-30 16:53:44,370 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:44,370 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:44,370 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:44,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:44,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:44,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:44,377 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:44,377 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:44,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:44,378 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:44,378 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:44,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:44,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:44,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:44,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:44,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:44,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6da90e3[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:44,380 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:44,380 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:44,385 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:44,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:44,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:44,385 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:44,385 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:44,386 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:44,386 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,386 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:44,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:44,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:44,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:44,388 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:44,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:44,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:44,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:44,393 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:44,394 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:44,394 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:44,394 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:44,394 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:44,396 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:44,396 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:44,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:44,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:44,396 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:44,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:44,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:44,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:44,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:44,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:44,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:44,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:44,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:44,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:44,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:44,399 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:44,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:44,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,402 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,407 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:44,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:44,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:44,407 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:44,408 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:44,408 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:44,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:44,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:44,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:44,408 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:44,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:44,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:44,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:44,412 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:44,412 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:44,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:44,513 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:44,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:44,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:44,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:44,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:44,518 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:44,518 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:44,519 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:44,519 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:44,520 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:44,520 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:44,523 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:44,523 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:44,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:44,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:44,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:44,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:44,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:44,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:44,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:44,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:44,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:44,527 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:44,527 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:44,528 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:44,529 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:44,529 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:44,529 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:44,529 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:44,529 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-30 16:53:44,530 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:44,530 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:44,530 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:44,530 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:44,530 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:44,530 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:44,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:44,775 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:44,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:44,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195284823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:44,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195284823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:44,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195284826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:44,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195284827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:44,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195284828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:44,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195284930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:44,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195284930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:44,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195284932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:44,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195284933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:44,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195284933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:44,992 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/b5f1f53a22e84398a2c81d9ba1348681 2023-03-30 16:53:45,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:45,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195285133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:45,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195285134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:45,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195285136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:45,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195285138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:45,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195285138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,418 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/870f5cd957474240bb711a0edf7f55b9 2023-03-30 16:53:45,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:45,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195285439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:45,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195285440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:45,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195285440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,445 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/6e0d61e1736446878914529cdbbe9d05 2023-03-30 16:53:45,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:45,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195285442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:45,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195285444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,454 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/b5f1f53a22e84398a2c81d9ba1348681 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/b5f1f53a22e84398a2c81d9ba1348681 2023-03-30 16:53:45,461 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/b5f1f53a22e84398a2c81d9ba1348681, entries=150, sequenceid=85, filesize=11.6 K 2023-03-30 16:53:45,463 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/870f5cd957474240bb711a0edf7f55b9 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/870f5cd957474240bb711a0edf7f55b9 2023-03-30 16:53:45,470 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/870f5cd957474240bb711a0edf7f55b9, entries=150, sequenceid=85, filesize=11.6 K 2023-03-30 16:53:45,474 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/6e0d61e1736446878914529cdbbe9d05 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/6e0d61e1736446878914529cdbbe9d05 2023-03-30 16:53:45,484 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/6e0d61e1736446878914529cdbbe9d05, entries=150, sequenceid=85, filesize=11.6 K 2023-03-30 16:53:45,486 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 4600ee2f6bff5cc4f7b8bf671d05894d in 957ms, sequenceid=85, compaction requested=true 2023-03-30 16:53:45,486 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:45,486 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:45,487 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:45,488 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:45,488 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,488 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:45,488 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:45,490 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,490 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:45,490 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:45,490 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:45,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:45,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:45,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@413076d0[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:45,492 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:45,492 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:45,495 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:45,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:45,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:45,495 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:45,495 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:45,495 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:45,495 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,495 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:45,497 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:45,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:45,498 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:45,498 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:45,498 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:45,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:45,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:45,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:45,502 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:45,502 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:45,502 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:45,502 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:45,502 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:45,504 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:45,504 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:45,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:45,504 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:45,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:45,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:45,505 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:45,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:45,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:45,507 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:45,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:45,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:45,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:45,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:45,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:45,509 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:45,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:45,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:45,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:45,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:45,511 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,511 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:45,511 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:45,511 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:45,510 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:45,511 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:45,611 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:45,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:45,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:45,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:45,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:45,616 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:45,616 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:45,617 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:45,617 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:45,617 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:45,617 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:45,624 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:45,624 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:45,625 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,625 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:45,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:45,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:45,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:45,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:45,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:45,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:45,626 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:45,626 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:45,627 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:45,629 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:45,629 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:45,629 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:45,629 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:45,629 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-30 16:53:45,630 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:45,630 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:45,630 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:45,630 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:45,630 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:45,630 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:45,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:45,945 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:45,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:45,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195285961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:45,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195285963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,968 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:45,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195285963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:45,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195285965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:45,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:45,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195285966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,055 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/448d6b0211304d54b05776103112d83d 2023-03-30 16:53:46,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195286068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195286069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195286069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195286072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195286072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,132 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/dfcbb9c1c7b44f299655775c25be503f 2023-03-30 16:53:46,177 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/10d26e892f814870a3f79786f9200a13 2023-03-30 16:53:46,186 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/448d6b0211304d54b05776103112d83d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/448d6b0211304d54b05776103112d83d 2023-03-30 16:53:46,195 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/448d6b0211304d54b05776103112d83d, entries=150, sequenceid=100, filesize=11.6 K 2023-03-30 16:53:46,199 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/dfcbb9c1c7b44f299655775c25be503f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/dfcbb9c1c7b44f299655775c25be503f 2023-03-30 16:53:46,207 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/dfcbb9c1c7b44f299655775c25be503f, entries=150, sequenceid=100, filesize=11.6 K 2023-03-30 16:53:46,212 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/10d26e892f814870a3f79786f9200a13 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/10d26e892f814870a3f79786f9200a13 2023-03-30 16:53:46,221 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/10d26e892f814870a3f79786f9200a13, entries=150, sequenceid=100, filesize=11.6 K 2023-03-30 16:53:46,223 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 4600ee2f6bff5cc4f7b8bf671d05894d in 594ms, sequenceid=100, compaction requested=true 2023-03-30 16:53:46,223 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:46,223 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:46,223 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:46,224 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:46,224 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,224 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:46,224 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:46,226 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:46,226 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:46,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:46,227 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:46,227 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:46,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:46,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:46,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e768c34[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:46,230 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:46,230 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:46,232 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:46,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:46,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:46,233 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:46,233 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:46,233 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:46,233 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,233 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:46,234 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:46,234 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:46,234 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:46,234 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:46,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:46,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:46,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:46,243 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:46,243 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:46,243 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:46,243 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:46,243 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:46,246 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:46,246 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:46,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:46,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:46,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:46,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:46,246 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:46,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:46,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:46,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:46,247 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,249 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,252 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:46,252 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:46,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:46,253 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,253 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:46,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:46,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:46,253 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:46,253 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:46,253 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,253 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:46,253 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:46,253 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:46,253 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:46,254 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:46,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:46,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:46,278 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-30 16:53:46,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:46,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:46,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:46,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:46,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:46,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:46,308 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/6b0f00f241a44e2c88779998a8be8271 2023-03-30 16:53:46,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195286295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195286306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195286308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195286310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195286310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,335 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/2f98af72f6084334b72e52b9824359e8 2023-03-30 16:53:46,354 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:46,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:46,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:46,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:46,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:46,360 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:46,360 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:46,363 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:46,363 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:46,365 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:46,365 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:46,367 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:46,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:46,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:46,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:46,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:46,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:46,369 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,369 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,369 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:46,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:46,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:46,373 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:46,373 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:46,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:46,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:46,375 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:46,375 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:46,375 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:46,377 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:46,377 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:46,377 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:46,377 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:46,377 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,377 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:46,377 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:46,380 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:46,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:46,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:46,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:46,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:46,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:46,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:46,381 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/00cc491da8724739aec0da7b6d84085c 2023-03-30 16:53:46,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:46,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:46,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f02bad3[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:46,383 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:46,383 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:46,384 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:46,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:46,385 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,385 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:46,386 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:46,386 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:46,386 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:46,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:46,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:46,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:46,389 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,391 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/6b0f00f241a44e2c88779998a8be8271 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/6b0f00f241a44e2c88779998a8be8271 2023-03-30 16:53:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:46,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:46,393 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:46,393 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:46,393 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:46,393 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:46,393 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:46,399 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:46,399 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:46,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:46,400 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:46,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:46,400 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/6b0f00f241a44e2c88779998a8be8271, entries=200, sequenceid=119, filesize=13.9 K 2023-03-30 16:53:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:46,401 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:46,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:46,403 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/2f98af72f6084334b72e52b9824359e8 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/2f98af72f6084334b72e52b9824359e8 2023-03-30 16:53:46,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:46,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:46,407 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,410 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,410 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:46,410 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:46,410 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:46,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:46,410 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:46,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:46,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:46,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:46,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:46,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:46,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:46,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:46,413 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:46,413 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:46,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:46,418 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/2f98af72f6084334b72e52b9824359e8, entries=150, sequenceid=119, filesize=11.6 K 2023-03-30 16:53:46,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195286413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195286413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,420 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/00cc491da8724739aec0da7b6d84085c as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/00cc491da8724739aec0da7b6d84085c 2023-03-30 16:53:46,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195286414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195286418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195286418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,429 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/00cc491da8724739aec0da7b6d84085c, entries=150, sequenceid=119, filesize=11.6 K 2023-03-30 16:53:46,430 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 4600ee2f6bff5cc4f7b8bf671d05894d in 153ms, sequenceid=119, compaction requested=true 2023-03-30 16:53:46,431 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:46,431 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:53:46,432 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 16:53:46,432 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 16:53:46,432 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:53:46,432 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:53:46,434 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61873 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 16:53:46,434 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/A is initiating minor compaction (all files) 2023-03-30 16:53:46,434 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59533 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 16:53:46,434 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/A in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:46,434 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/B is initiating minor compaction (all files) 2023-03-30 16:53:46,434 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/2fe13d19e14e49c49f5e1505655faa14, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/5d9e9067be924258b1f917729735c8ff, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/b5f1f53a22e84398a2c81d9ba1348681, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/448d6b0211304d54b05776103112d83d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/6b0f00f241a44e2c88779998a8be8271] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=60.4 K 2023-03-30 16:53:46,434 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/B in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:46,434 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/ba389fa14dde4064a070b904b13a7509, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/acecd5442a3c4e5dba63a830513000a5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/870f5cd957474240bb711a0edf7f55b9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/dfcbb9c1c7b44f299655775c25be503f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/2f98af72f6084334b72e52b9824359e8] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=58.1 K 2023-03-30 16:53:46,435 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 2fe13d19e14e49c49f5e1505655faa14, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680195220794 2023-03-30 16:53:46,435 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting ba389fa14dde4064a070b904b13a7509, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680195220794 2023-03-30 16:53:46,438 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 5d9e9067be924258b1f917729735c8ff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680195221462 2023-03-30 16:53:46,438 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting acecd5442a3c4e5dba63a830513000a5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680195221462 2023-03-30 16:53:46,439 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting b5f1f53a22e84398a2c81d9ba1348681, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680195223646 2023-03-30 16:53:46,439 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 870f5cd957474240bb711a0edf7f55b9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680195223646 2023-03-30 16:53:46,439 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 448d6b0211304d54b05776103112d83d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680195224818 2023-03-30 16:53:46,441 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting dfcbb9c1c7b44f299655775c25be503f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680195224818 2023-03-30 16:53:46,441 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 6b0f00f241a44e2c88779998a8be8271, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680195225957 2023-03-30 16:53:46,442 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 2f98af72f6084334b72e52b9824359e8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680195225961 2023-03-30 16:53:46,464 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#A#compaction#24 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:53:46,469 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#B#compaction#25 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:53:46,492 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/0b23d0496553497cba38d70407bb16b1 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/0b23d0496553497cba38d70407bb16b1 2023-03-30 16:53:46,501 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/A of 4600ee2f6bff5cc4f7b8bf671d05894d into 0b23d0496553497cba38d70407bb16b1(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:53:46,501 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:46,501 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/A, priority=11, startTime=1680195226431; duration=0sec 2023-03-30 16:53:46,501 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:53:46,501 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 16:53:46,504 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59533 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 16:53:46,504 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/C is initiating minor compaction (all files) 2023-03-30 16:53:46,505 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/C in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:46,505 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/b484898c087d4f14bc44df334024faf5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/26d909c968cb41958c7a053c0e442382, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/6e0d61e1736446878914529cdbbe9d05, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/10d26e892f814870a3f79786f9200a13, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/00cc491da8724739aec0da7b6d84085c] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=58.1 K 2023-03-30 16:53:46,508 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting b484898c087d4f14bc44df334024faf5, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1680195220794 2023-03-30 16:53:46,512 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 26d909c968cb41958c7a053c0e442382, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1680195221462 2023-03-30 16:53:46,513 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:46,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:46,516 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 6e0d61e1736446878914529cdbbe9d05, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680195223646 2023-03-30 16:53:46,517 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 10d26e892f814870a3f79786f9200a13, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1680195224818 2023-03-30 16:53:46,518 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 00cc491da8724739aec0da7b6d84085c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680195225961 2023-03-30 16:53:46,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:46,527 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/f0eda55a7c2a4759a3dc770a3247d10b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/f0eda55a7c2a4759a3dc770a3247d10b 2023-03-30 16:53:46,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:46,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:46,531 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:46,531 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:46,534 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:46,534 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:46,538 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:46,538 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:46,539 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:46,539 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:46,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:46,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:46,540 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,540 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:46,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:46,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:46,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:46,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:46,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:46,542 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:46,542 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:46,543 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:46,543 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:46,543 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:46,543 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:46,543 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:46,544 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-30 16:53:46,544 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:46,544 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:46,544 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:46,544 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:46,545 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:46,545 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:46,555 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/B of 4600ee2f6bff5cc4f7b8bf671d05894d into f0eda55a7c2a4759a3dc770a3247d10b(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:53:46,555 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:46,555 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/B, priority=11, startTime=1680195226432; duration=0sec 2023-03-30 16:53:46,555 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:53:46,562 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#C#compaction#27 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:53:46,573 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/60a786b3c1fd46f8aec2670522f3e4a7 2023-03-30 16:53:46,596 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/09730aa06f3844c1b4f2a22bcacbf2e4 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/09730aa06f3844c1b4f2a22bcacbf2e4 2023-03-30 16:53:46,607 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/C of 4600ee2f6bff5cc4f7b8bf671d05894d into 09730aa06f3844c1b4f2a22bcacbf2e4(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:53:46,607 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:46,607 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/C, priority=11, startTime=1680195226432; duration=0sec 2023-03-30 16:53:46,607 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:53:46,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:46,625 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:46,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195286654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195286655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195286658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195286659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195286660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195286763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195286763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195286764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195286767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195286767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195286967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195286969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195286970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195286972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:46,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:46,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195286973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,037 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/261cda4e0e2d4e40aed7d1e2c8d5e7f8 2023-03-30 16:53:47,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:47,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195287271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:47,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195287272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:47,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195287273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:47,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195287274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:47,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195287278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,464 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/e54d4fdd9ede4b7caa0c5ddad6fa258b 2023-03-30 16:53:47,479 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/60a786b3c1fd46f8aec2670522f3e4a7 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/60a786b3c1fd46f8aec2670522f3e4a7 2023-03-30 16:53:47,491 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/60a786b3c1fd46f8aec2670522f3e4a7, entries=150, sequenceid=133, filesize=11.7 K 2023-03-30 16:53:47,494 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/261cda4e0e2d4e40aed7d1e2c8d5e7f8 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/261cda4e0e2d4e40aed7d1e2c8d5e7f8 2023-03-30 16:53:47,504 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/261cda4e0e2d4e40aed7d1e2c8d5e7f8, entries=150, sequenceid=133, filesize=11.7 K 2023-03-30 16:53:47,505 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/e54d4fdd9ede4b7caa0c5ddad6fa258b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/e54d4fdd9ede4b7caa0c5ddad6fa258b 2023-03-30 16:53:47,521 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/e54d4fdd9ede4b7caa0c5ddad6fa258b, entries=150, sequenceid=133, filesize=11.7 K 2023-03-30 16:53:47,524 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 4600ee2f6bff5cc4f7b8bf671d05894d in 980ms, sequenceid=133, compaction requested=false 2023-03-30 16:53:47,524 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:47,524 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:47,524 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:47,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:47,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,532 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:47,532 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:47,535 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:47,539 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:47,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:47,539 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:47,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d8d6458[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:47,539 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:47,539 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:47,540 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:47,541 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:47,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:47,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:47,541 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:47,542 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:47,542 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:47,543 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,543 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:47,545 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:47,545 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,545 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:47,545 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:47,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:47,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:47,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:47,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:47,555 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:47,555 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:47,555 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:47,555 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:47,555 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:47,557 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:47,557 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:47,557 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:47,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:47,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:47,558 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:47,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:47,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:47,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,560 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:47,564 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,564 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:47,564 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:47,564 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,564 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:47,564 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:47,564 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:47,564 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:47,564 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:47,564 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:47,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:47,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:47,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:47,570 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:47,570 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:47,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:47,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:47,670 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:47,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:47,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:47,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:47,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:47,677 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:47,677 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:47,678 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:47,678 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:47,678 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:47,679 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:47,680 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:47,680 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:47,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:47,681 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,681 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:47,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:47,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:47,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:47,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:47,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:47,684 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:47,684 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:47,687 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:47,687 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:47,688 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-30 16:53:47,688 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:47,688 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:47,689 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:47,689 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:47,689 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:47,689 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:47,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:47,777 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:47,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:47,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195287787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:47,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195287787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:47,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195287787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:47,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195287788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:47,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195287789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:47,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195287892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:47,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195287893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:47,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195287893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:47,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195287893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:47,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:47,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195287893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:48,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195288095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:48,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195288098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:48,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195288099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:48,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195288099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:48,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195288100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,111 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/00b8445b390e434dbbf2f159a66adcac 2023-03-30 16:53:48,152 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/3c5eb3ff345a4c02a1c3aff97bba8a46 2023-03-30 16:53:48,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:48,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195288398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:48,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195288402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:48,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195288405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:48,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195288406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:48,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195288406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,582 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/5da908072b584a7d8693f416ec29fca8 2023-03-30 16:53:48,591 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/00b8445b390e434dbbf2f159a66adcac as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/00b8445b390e434dbbf2f159a66adcac 2023-03-30 16:53:48,599 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/00b8445b390e434dbbf2f159a66adcac, entries=150, sequenceid=155, filesize=11.8 K 2023-03-30 16:53:48,603 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/3c5eb3ff345a4c02a1c3aff97bba8a46 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/3c5eb3ff345a4c02a1c3aff97bba8a46 2023-03-30 16:53:48,613 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/3c5eb3ff345a4c02a1c3aff97bba8a46, entries=150, sequenceid=155, filesize=11.8 K 2023-03-30 16:53:48,617 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/5da908072b584a7d8693f416ec29fca8 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/5da908072b584a7d8693f416ec29fca8 2023-03-30 16:53:48,627 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/5da908072b584a7d8693f416ec29fca8, entries=150, sequenceid=155, filesize=11.8 K 2023-03-30 16:53:48,628 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 4600ee2f6bff5cc4f7b8bf671d05894d in 940ms, sequenceid=155, compaction requested=true 2023-03-30 16:53:48,628 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:48,629 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:48,635 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:48,635 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,635 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:48,635 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:48,637 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:48,638 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:48,638 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:48,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:48,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:48,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fc905cf[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:48,640 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:48,640 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:48,642 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:48,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:48,642 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,643 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:48,644 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,644 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:48,644 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,644 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:48,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:48,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:48,655 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:48,655 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:48,655 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:48,656 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:48,656 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:48,658 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:48,658 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:48,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:48,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:48,659 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:48,660 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:48,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:48,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:48,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:48,673 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:48,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:48,683 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,683 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:48,683 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:48,683 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:48,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:48,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:48,683 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:48,683 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:48,694 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,694 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:48,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:48,694 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:48,694 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:48,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:48,694 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:48,694 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:48,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:48,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:48,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:48,794 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:48,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:48,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:48,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:48,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:48,824 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:48,825 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:48,825 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:48,825 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:48,825 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:48,825 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:48,828 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,829 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:48,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:48,829 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,829 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:48,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:48,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:48,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:48,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:48,840 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:48,840 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:48,840 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:48,844 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:48,844 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:48,844 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:48,845 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:48,845 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-30 16:53:48,846 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:48,846 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:48,846 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:48,846 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:48,846 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:48,846 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:48,882 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/520cf7a76ff24bc785d8e9890feebad6 2023-03-30 16:53:48,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:48,910 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:48,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:48,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195288957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:48,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195288957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:48,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195288958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:48,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195288963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:48,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:48,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195288965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195289067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195289067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195289067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195289071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195289072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195289269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195289273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195289273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195289275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195289276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,303 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/2e9ea38d207f47588284da2da2a46172 2023-03-30 16:53:49,317 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/3f3d18ef30184aedbe3c557358e76f2f 2023-03-30 16:53:49,324 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/520cf7a76ff24bc785d8e9890feebad6 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/520cf7a76ff24bc785d8e9890feebad6 2023-03-30 16:53:49,351 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/520cf7a76ff24bc785d8e9890feebad6, entries=150, sequenceid=167, filesize=11.8 K 2023-03-30 16:53:49,352 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/2e9ea38d207f47588284da2da2a46172 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/2e9ea38d207f47588284da2da2a46172 2023-03-30 16:53:49,358 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/2e9ea38d207f47588284da2da2a46172, entries=150, sequenceid=167, filesize=11.8 K 2023-03-30 16:53:49,360 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/3f3d18ef30184aedbe3c557358e76f2f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/3f3d18ef30184aedbe3c557358e76f2f 2023-03-30 16:53:49,369 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/3f3d18ef30184aedbe3c557358e76f2f, entries=150, sequenceid=167, filesize=11.8 K 2023-03-30 16:53:49,371 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4600ee2f6bff5cc4f7b8bf671d05894d in 526ms, sequenceid=167, compaction requested=true 2023-03-30 16:53:49,371 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:49,371 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:49,371 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:49,373 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:49,373 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,373 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:49,373 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:49,375 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,375 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:49,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:49,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:49,376 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:49,376 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:49,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:49,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75e4372e[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:49,377 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:49,377 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:49,379 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:49,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:49,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:49,379 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:49,379 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:49,379 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:49,379 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,379 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:49,380 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:49,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:49,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:49,383 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:49,383 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:49,383 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:49,383 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:49,383 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:49,386 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:49,386 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:49,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:49,386 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:49,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:49,386 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:49,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:49,388 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:49,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:49,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,394 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:49,394 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,394 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:49,394 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:49,394 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:49,394 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:49,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:49,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:49,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:49,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:49,394 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,394 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:49,394 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:49,395 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:49,395 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:49,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:49,395 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:49,495 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:49,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:49,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:49,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:49,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:49,500 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:49,500 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:49,500 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:49,500 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:49,500 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:49,500 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:49,502 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:49,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:49,502 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:49,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:49,502 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,503 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:49,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:49,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:49,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:49,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:49,504 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:49,504 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:49,504 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:49,504 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:49,504 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:49,504 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:49,504 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:49,504 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-30 16:53:49,505 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:49,505 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:49,505 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:49,505 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:49,505 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:49,505 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:49,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:49,579 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:49,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195289592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195289592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195289597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195289598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195289599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195289699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195289700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195289705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195289705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195289705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195289906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195289906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195289909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:49,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195289910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195289911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:49,917 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/fdddefd957304fefbd84d5745910e2c7 2023-03-30 16:53:50,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:50,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:50,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195290209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195290212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:50,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195290220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:50,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195290220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:50,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195290220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,342 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/86f9209ec63b48c7a1b65bd993ec8823 2023-03-30 16:53:50,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:50,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195290718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:50,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195290719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:50,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195290725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:50,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195290728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:50,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195290728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,761 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/03f2c4a20209410383c65822e5c5b48b 2023-03-30 16:53:50,767 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/fdddefd957304fefbd84d5745910e2c7 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/fdddefd957304fefbd84d5745910e2c7 2023-03-30 16:53:50,773 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/fdddefd957304fefbd84d5745910e2c7, entries=150, sequenceid=187, filesize=11.8 K 2023-03-30 16:53:50,774 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/86f9209ec63b48c7a1b65bd993ec8823 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/86f9209ec63b48c7a1b65bd993ec8823 2023-03-30 16:53:50,780 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/86f9209ec63b48c7a1b65bd993ec8823, entries=150, sequenceid=187, filesize=11.8 K 2023-03-30 16:53:50,781 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/03f2c4a20209410383c65822e5c5b48b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/03f2c4a20209410383c65822e5c5b48b 2023-03-30 16:53:50,786 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/03f2c4a20209410383c65822e5c5b48b, entries=150, sequenceid=187, filesize=11.8 K 2023-03-30 16:53:50,787 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=73.80 KB/75570 for 4600ee2f6bff5cc4f7b8bf671d05894d in 1283ms, sequenceid=187, compaction requested=true 2023-03-30 16:53:50,787 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:50,787 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:50,788 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:50,789 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:50,789 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,789 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:50,789 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:50,791 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,791 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:50,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:50,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:50,791 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:50,791 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f7acf63[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:50,793 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:50,793 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:50,794 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:50,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:50,794 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:50,794 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:50,794 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,795 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:50,794 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:50,798 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:50,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:50,800 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:50,801 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:50,801 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:50,801 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:50,801 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:50,802 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:50,802 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:50,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:50,802 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:50,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:50,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:50,803 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:50,804 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:50,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:50,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:50,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:50,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:50,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:50,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:50,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:50,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:50,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:50,808 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:50,808 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:50,808 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:50,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:50,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:50,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:50,908 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:50,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:50,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:50,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:50,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:50,916 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:50,916 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:50,916 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:50,916 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:50,917 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:50,917 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:50,922 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,922 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:50,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:50,923 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:50,923 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:50,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:50,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:50,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:50,924 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:50,924 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:50,924 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:50,925 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:50,925 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:50,925 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-30 16:53:50,925 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:50,925 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:50,925 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:50,925 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:50,925 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:50,925 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:51,352 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/237b877eac3e47399e04dd4611c30ac2 2023-03-30 16:53:51,372 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/640fa9c6d2d943b59fc840c092cf340b 2023-03-30 16:53:51,385 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/810afd450dad4541875bdd16da98f66a 2023-03-30 16:53:51,394 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/237b877eac3e47399e04dd4611c30ac2 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/237b877eac3e47399e04dd4611c30ac2 2023-03-30 16:53:51,400 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/237b877eac3e47399e04dd4611c30ac2, entries=150, sequenceid=201, filesize=11.8 K 2023-03-30 16:53:51,401 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/640fa9c6d2d943b59fc840c092cf340b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/640fa9c6d2d943b59fc840c092cf340b 2023-03-30 16:53:51,408 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/640fa9c6d2d943b59fc840c092cf340b, entries=150, sequenceid=201, filesize=11.8 K 2023-03-30 16:53:51,410 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/810afd450dad4541875bdd16da98f66a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/810afd450dad4541875bdd16da98f66a 2023-03-30 16:53:51,420 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/810afd450dad4541875bdd16da98f66a, entries=150, sequenceid=201, filesize=11.8 K 2023-03-30 16:53:51,421 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 4600ee2f6bff5cc4f7b8bf671d05894d in 496ms, sequenceid=201, compaction requested=true 2023-03-30 16:53:51,421 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:51,421 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:51,421 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:51,422 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:51,422 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,422 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:51,422 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:51,424 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,424 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:51,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:51,425 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,425 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:51,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:51,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:51,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:51,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5933c370[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:51,427 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:51,427 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,430 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,430 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:51,430 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:51,430 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:51,431 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,431 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:51,432 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:51,432 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,432 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:51,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:51,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:51,432 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:51,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:51,437 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:51,437 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:51,437 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:51,437 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:51,437 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:51,439 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,439 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:51,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,439 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:51,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:51,440 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:51,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,442 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,446 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,446 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:51,446 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:51,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:51,446 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:51,446 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:51,446 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,446 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,446 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,446 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:51,446 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:51,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:51,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:51,447 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:51,447 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:51,547 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:51,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:51,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:51,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:51,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:51,551 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:51,551 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:51,551 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:51,551 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:51,552 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,552 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:51,553 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:51,553 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:51,554 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,554 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:51,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:51,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:51,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:51,555 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:51,555 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:51,555 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:51,556 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:51,556 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:51,556 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:51,556 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:51,556 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:51,556 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:51,556 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:51,556 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:51,556 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,556 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:51,556 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:51,558 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:51,559 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,559 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:51,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:51,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ed342e4[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:51,567 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:51,567 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,569 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,569 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:51,569 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:51,569 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:51,570 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,570 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:51,572 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,572 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:51,572 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:51,572 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:51,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:51,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:51,577 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:51,577 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:51,577 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:51,577 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:51,577 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:51,579 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:51,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:51,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,581 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,584 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,585 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,585 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:51,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:51,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,592 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:51,592 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:51,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:51,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:51,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:51,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:51,593 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:51,593 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:51,592 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:51,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:51,592 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,594 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:51,594 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:51,594 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,594 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,594 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,594 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,693 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:51,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:51,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:51,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:51,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:51,696 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:51,696 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:51,696 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:51,696 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:51,697 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,697 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:51,698 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:51,698 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:51,699 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:51,699 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:51,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:51,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:51,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:51,700 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:51,700 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:51,700 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:51,701 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:51,701 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:51,701 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:51,701 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:51,701 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:51,701 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:51,701 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:51,701 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:51,701 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,701 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:51,701 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:51,704 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:51,708 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:51,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:51,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ada4357[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:51,710 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:51,710 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,711 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,712 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:51,712 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:51,712 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:51,712 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,712 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:51,713 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:51,713 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:51,713 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:51,715 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:51,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:51,720 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:51,720 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:51,720 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:51,720 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:51,720 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:51,724 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,724 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,724 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:51,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:51,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:51,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,725 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:51,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:51,729 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:51,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:51,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:51,733 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:51,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:51,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:51,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:51,734 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:51,734 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:51,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:51,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:51,736 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,736 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:51,736 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:51,736 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,736 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,736 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,736 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:51,743 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:53:51,743 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:51,743 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:51,743 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:51,744 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:51,744 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:51,744 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:51,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:51,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195291779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:51,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195291781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:51,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195291782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195231774, max timestamp=1680195231782Num uniques 859; 2023-03-30 16:53:51,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195231774, max timestamp=1680195231782Num uniques 859; 2023-03-30 16:53:51,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:53:51,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:53:51,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195231774, max timestamp=1680195231782Num uniques 859; 2023-03-30 16:53:51,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:53:51,834 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:51,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:51,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:51,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:51,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:51,852 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:51,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:51,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:51,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:51,855 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,855 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:51,857 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,857 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,857 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:51,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:51,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:51,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:51,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:51,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:51,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:51,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:51,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:51,868 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:51,868 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:51,868 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:51,868 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:51,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:51,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:51,869 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,869 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:51,869 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:51,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:51,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41762 deadline: 1680195291865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,870 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,870 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:51,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:51,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:51,877 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,877 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:51,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:51,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:51,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:51,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@633284b8[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:51,879 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:51,879 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:51,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41792 deadline: 1680195291876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,882 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,882 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:51,882 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:51,882 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:51,884 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,884 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:51,884 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:51,884 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:51,886 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:51,886 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:51,886 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:51,886 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:51,886 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:51,886 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:51,886 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:51,889 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,889 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:51,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,889 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:51,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:51,892 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,895 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:51,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195291895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:51,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:51,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:51,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:51,899 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:51,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195291896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:51,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:51,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:51,900 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:51,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:51,900 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:51,900 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,900 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,900 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:51,900 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:51,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:51,901 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:51,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195291896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,901 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:51,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:51,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195291977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:51,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:51,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41792 deadline: 1680195291982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,001 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:52,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:52,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:52,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:52,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:52,005 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:52,005 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:52,005 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:52,005 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:52,005 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:52,005 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:52,007 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:52,007 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:52,008 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:52,008 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:52,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:52,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:52,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:52,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:52,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:52,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:52,009 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:52,009 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:52,009 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:52,009 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:52,009 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:52,010 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:52,010 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:52,010 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:52,010 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:52,010 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:52,010 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:52,010 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,010 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:52,010 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:52,012 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:52,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:52,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:52,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:52,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:52,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@245de3ed[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:52,014 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:52,014 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:52,015 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:52,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:52,016 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,016 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:52,016 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:52,016 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:52,016 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:52,021 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:52,021 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:52,021 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:52,022 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:52,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:52,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:52,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:52,026 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:52,026 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:52,027 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:52,027 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:52,027 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:52,029 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:52,029 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:52,029 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:52,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:52,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:52,030 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:52,032 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:52,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:52,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,035 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,035 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:52,035 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:52,035 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,035 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:52,035 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:52,036 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:52,036 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:52,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:52,036 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:52,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:52,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:52,036 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:52,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:52,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:52,036 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:52,036 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:52,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:52,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:52,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:52,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195292100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:52,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195292102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:52,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41828 deadline: 1680195292103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,136 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:52,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:52,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:52,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:52,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:52,141 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:52,141 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:52,141 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:52,141 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:52,141 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:52,141 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:52,144 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:52,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:52,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:52,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:52,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:52,145 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:52,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:52,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:52,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:52,145 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,145 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:52,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:52,146 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:52,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:52,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:52,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:52,146 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:52,146 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:52,146 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:52,146 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:52,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:52,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:52,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:52,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:52,150 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,150 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:52,150 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:52,150 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:52,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:52,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60dd41e0[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:52,152 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:52,152 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:52,153 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:52,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:52,153 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:52,153 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:52,153 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:52,153 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,153 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:52,156 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,156 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:52,156 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:52,156 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:52,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:52,158 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:52,158 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:52,158 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:52,158 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:52,158 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:52,160 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:52,160 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:52,160 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:52,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:52,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:52,161 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,162 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,165 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,165 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:52,165 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:52,165 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,165 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:52,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:52,166 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:52,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:52,166 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:52,165 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:52,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:52,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:52,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:52,166 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:52,166 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:52,166 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:52,166 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:52,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:52,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:52,170 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/dc6e78c02b1d41abbb512cfb0fcd20eb 2023-03-30 16:53:52,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:52,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195292182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,185 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/a7b482e81c524d679d4b9e739b276611 2023-03-30 16:53:52,188 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:52,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41792 deadline: 1680195292186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,197 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/affd0070495e49a19172de13a0652541 2023-03-30 16:53:52,205 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/dc6e78c02b1d41abbb512cfb0fcd20eb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/dc6e78c02b1d41abbb512cfb0fcd20eb 2023-03-30 16:53:52,210 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/dc6e78c02b1d41abbb512cfb0fcd20eb, entries=150, sequenceid=213, filesize=11.8 K 2023-03-30 16:53:52,210 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-30 16:53:52,211 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/a7b482e81c524d679d4b9e739b276611 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/a7b482e81c524d679d4b9e739b276611 2023-03-30 16:53:52,217 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/a7b482e81c524d679d4b9e739b276611, entries=150, sequenceid=213, filesize=11.8 K 2023-03-30 16:53:52,218 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/affd0070495e49a19172de13a0652541 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/affd0070495e49a19172de13a0652541 2023-03-30 16:53:52,223 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/affd0070495e49a19172de13a0652541, entries=150, sequenceid=213, filesize=11.8 K 2023-03-30 16:53:52,224 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 4600ee2f6bff5cc4f7b8bf671d05894d in 481ms, sequenceid=213, compaction requested=true 2023-03-30 16:53:52,224 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:52,224 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:53:52,224 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-30 16:53:52,224 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-30 16:53:52,224 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:53:52,225 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:53:52,228 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84326 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-30 16:53:52,229 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84326 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-30 16:53:52,229 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/A is initiating minor compaction (all files) 2023-03-30 16:53:52,229 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/B is initiating minor compaction (all files) 2023-03-30 16:53:52,229 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/A in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:52,229 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/B in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:52,229 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/0b23d0496553497cba38d70407bb16b1, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/60a786b3c1fd46f8aec2670522f3e4a7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/00b8445b390e434dbbf2f159a66adcac, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/520cf7a76ff24bc785d8e9890feebad6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/fdddefd957304fefbd84d5745910e2c7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/237b877eac3e47399e04dd4611c30ac2, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/dc6e78c02b1d41abbb512cfb0fcd20eb] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=82.3 K 2023-03-30 16:53:52,229 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/f0eda55a7c2a4759a3dc770a3247d10b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/261cda4e0e2d4e40aed7d1e2c8d5e7f8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/3c5eb3ff345a4c02a1c3aff97bba8a46, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/2e9ea38d207f47588284da2da2a46172, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/86f9209ec63b48c7a1b65bd993ec8823, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/640fa9c6d2d943b59fc840c092cf340b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/a7b482e81c524d679d4b9e739b276611] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=82.3 K 2023-03-30 16:53:52,229 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 0b23d0496553497cba38d70407bb16b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680195225961 2023-03-30 16:53:52,230 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting f0eda55a7c2a4759a3dc770a3247d10b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680195225961 2023-03-30 16:53:52,232 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 261cda4e0e2d4e40aed7d1e2c8d5e7f8, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680195226289 2023-03-30 16:53:52,232 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 60a786b3c1fd46f8aec2670522f3e4a7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680195226289 2023-03-30 16:53:52,233 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 3c5eb3ff345a4c02a1c3aff97bba8a46, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680195226650 2023-03-30 16:53:52,233 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 00b8445b390e434dbbf2f159a66adcac, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680195226650 2023-03-30 16:53:52,233 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 2e9ea38d207f47588284da2da2a46172, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680195227783 2023-03-30 16:53:52,233 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 520cf7a76ff24bc785d8e9890feebad6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680195227783 2023-03-30 16:53:52,234 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting fdddefd957304fefbd84d5745910e2c7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1680195228956 2023-03-30 16:53:52,234 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 86f9209ec63b48c7a1b65bd993ec8823, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1680195228956 2023-03-30 16:53:52,235 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 237b877eac3e47399e04dd4611c30ac2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680195229584 2023-03-30 16:53:52,235 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 640fa9c6d2d943b59fc840c092cf340b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680195229584 2023-03-30 16:53:52,236 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting dc6e78c02b1d41abbb512cfb0fcd20eb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680195231734 2023-03-30 16:53:52,236 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting a7b482e81c524d679d4b9e739b276611, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680195231734 2023-03-30 16:53:52,252 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#B#compaction#45 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:53:52,252 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#A#compaction#46 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:53:52,266 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:52,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:52,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:52,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:52,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:52,270 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:52,270 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:52,270 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:52,270 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:52,270 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:52,270 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:52,272 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,272 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:52,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:52,272 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,272 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:52,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:52,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:52,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:52,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:52,276 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:52,276 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:52,276 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:52,280 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:52,280 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:52,280 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-30 16:53:52,280 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:52,281 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:53:52,281 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:52,281 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:53:52,281 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:52,281 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:53:52,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:52,410 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:52,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:52,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41828 deadline: 1680195292423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:52,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195292424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:52,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195292427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:52,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195292487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:52,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41792 deadline: 1680195292489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:52,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41828 deadline: 1680195292529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:52,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41798 deadline: 1680195292529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:52,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195292531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,676 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/084e3dad8cb545738861f2825de6deb0 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/084e3dad8cb545738861f2825de6deb0 2023-03-30 16:53:52,679 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/9c61a835c1cf4827b9a17db31ba5534d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/9c61a835c1cf4827b9a17db31ba5534d 2023-03-30 16:53:52,689 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/A of 4600ee2f6bff5cc4f7b8bf671d05894d into 9c61a835c1cf4827b9a17db31ba5534d(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:53:52,689 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:52,689 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/A, priority=9, startTime=1680195232224; duration=0sec 2023-03-30 16:53:52,689 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:53:52,689 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-30 16:53:52,689 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/B of 4600ee2f6bff5cc4f7b8bf671d05894d into 084e3dad8cb545738861f2825de6deb0(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:53:52,689 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:52,689 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/B, priority=9, startTime=1680195232224; duration=0sec 2023-03-30 16:53:52,689 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:53:52,692 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84326 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-30 16:53:52,693 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/C is initiating minor compaction (all files) 2023-03-30 16:53:52,693 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/C in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:52,693 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/09730aa06f3844c1b4f2a22bcacbf2e4, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/e54d4fdd9ede4b7caa0c5ddad6fa258b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/5da908072b584a7d8693f416ec29fca8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/3f3d18ef30184aedbe3c557358e76f2f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/03f2c4a20209410383c65822e5c5b48b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/810afd450dad4541875bdd16da98f66a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/affd0070495e49a19172de13a0652541] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=82.3 K 2023-03-30 16:53:52,693 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 09730aa06f3844c1b4f2a22bcacbf2e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680195225961 2023-03-30 16:53:52,694 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting e54d4fdd9ede4b7caa0c5ddad6fa258b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1680195226289 2023-03-30 16:53:52,695 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 5da908072b584a7d8693f416ec29fca8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680195226650 2023-03-30 16:53:52,695 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 3f3d18ef30184aedbe3c557358e76f2f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680195227783 2023-03-30 16:53:52,696 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 03f2c4a20209410383c65822e5c5b48b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1680195228956 2023-03-30 16:53:52,696 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 810afd450dad4541875bdd16da98f66a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680195229584 2023-03-30 16:53:52,697 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting affd0070495e49a19172de13a0652541, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680195231734 2023-03-30 16:53:52,699 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/9a991396fbf24827a335f893269eab57 2023-03-30 16:53:52,718 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#C#compaction#49 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:53:52,733 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/635d157ae0394d94911bf0a13135ce84 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/635d157ae0394d94911bf0a13135ce84 2023-03-30 16:53:52,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:52,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41828 deadline: 1680195292733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:52,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41798 deadline: 1680195292736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,742 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/C of 4600ee2f6bff5cc4f7b8bf671d05894d into 635d157ae0394d94911bf0a13135ce84(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:53:52,742 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:52,742 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/C, priority=9, startTime=1680195232225; duration=0sec 2023-03-30 16:53:52,743 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:53:52,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:52,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41814 deadline: 1680195292742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:52,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195292993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:52,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:52,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41792 deadline: 1680195292995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:53,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41828 deadline: 1680195293037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,046 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:53,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41798 deadline: 1680195293042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:53,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41814 deadline: 1680195293046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,116 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/ba9c6b4839e04ea0a87814c455351f58 2023-03-30 16:53:53,530 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/a0809f3e316e401e99c3de62e3d6cd69 2023-03-30 16:53:53,537 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/9a991396fbf24827a335f893269eab57 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/9a991396fbf24827a335f893269eab57 2023-03-30 16:53:53,543 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/9a991396fbf24827a335f893269eab57, entries=150, sequenceid=249, filesize=11.8 K 2023-03-30 16:53:53,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:53,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41828 deadline: 1680195293542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,545 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/ba9c6b4839e04ea0a87814c455351f58 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/ba9c6b4839e04ea0a87814c455351f58 2023-03-30 16:53:53,551 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/ba9c6b4839e04ea0a87814c455351f58, entries=150, sequenceid=249, filesize=11.8 K 2023-03-30 16:53:53,553 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/a0809f3e316e401e99c3de62e3d6cd69 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/a0809f3e316e401e99c3de62e3d6cd69 2023-03-30 16:53:53,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:53,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41798 deadline: 1680195293551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:53,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41814 deadline: 1680195293554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,567 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/a0809f3e316e401e99c3de62e3d6cd69, entries=150, sequenceid=249, filesize=11.8 K 2023-03-30 16:53:53,570 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 4600ee2f6bff5cc4f7b8bf671d05894d in 1289ms, sequenceid=249, compaction requested=false 2023-03-30 16:53:53,570 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:53,570 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:53,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:53,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:53,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:53,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:53,572 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:53,572 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:53,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:53,573 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:53,573 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:53,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:53,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:53,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:53,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38457657[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:53,574 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:53,575 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:53,577 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:53,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:53,577 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:53,577 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:53,577 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:53,577 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,578 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:53,579 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:53,579 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:53,579 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:53,580 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:53,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:53,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:53,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:53,585 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:53,585 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:53,585 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:53,585 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:53,585 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:53,587 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:53,587 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:53,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:53,587 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:53,588 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:53,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:53,597 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,600 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:53,600 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:53,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:53,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:53,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:53,600 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:53,600 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,601 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:53,601 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:53,601 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:53,601 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:53,601 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,601 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:53,601 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:53,601 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:53,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:53,701 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:53,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:53,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:53,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:53,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:53,705 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:53,705 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:53,705 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:53,705 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:53,706 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:53,706 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:53,707 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:53,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:53,707 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:53,708 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:53,708 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:53,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:53,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:53,716 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:53,716 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:53,716 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:53,716 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:53,716 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:53,716 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:53,716 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:53,716 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:53,716 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:53,717 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:53:53,717 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:53,717 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:53,717 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:53,717 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:53,717 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:53,717 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:54,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:54,004 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:54,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195234017, max timestamp=1680195234037Num uniques 859; 2023-03-30 16:53:54,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:53:54,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195234017, max timestamp=1680195234037Num uniques 859; 2023-03-30 16:53:54,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:53:54,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195234017, max timestamp=1680195234037Num uniques 859; 2023-03-30 16:53:54,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:53:54,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680195234061, max timestamp=1680195234073Num uniques 1018; 2023-03-30 16:53:54,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-30 16:53:54,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680195234061, max timestamp=1680195234073Num uniques 1018; 2023-03-30 16:53:54,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680195234061, max timestamp=1680195234073Num uniques 1018; 2023-03-30 16:53:54,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-30 16:53:54,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-30 16:53:54,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:54,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41792 deadline: 1680195294113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:54,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195294114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,134 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/89c391536c5c4295bcd4a7aad2bc95e1 2023-03-30 16:53:54,160 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/6832861d49be49e09090bb75cf800d4c 2023-03-30 16:53:54,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:54,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195294222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:54,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41792 deadline: 1680195294223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:54,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41792 deadline: 1680195294427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:54,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195294427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:54,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41828 deadline: 1680195294550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:54,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41814 deadline: 1680195294557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:54,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41798 deadline: 1680195294560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,638 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/cd1373f64fb94637ac2f8fd44472d88b 2023-03-30 16:53:54,645 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/89c391536c5c4295bcd4a7aad2bc95e1 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/89c391536c5c4295bcd4a7aad2bc95e1 2023-03-30 16:53:54,673 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/89c391536c5c4295bcd4a7aad2bc95e1, entries=150, sequenceid=262, filesize=11.8 K 2023-03-30 16:53:54,674 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/6832861d49be49e09090bb75cf800d4c as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/6832861d49be49e09090bb75cf800d4c 2023-03-30 16:53:54,683 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/6832861d49be49e09090bb75cf800d4c, entries=150, sequenceid=262, filesize=11.8 K 2023-03-30 16:53:54,684 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/cd1373f64fb94637ac2f8fd44472d88b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/cd1373f64fb94637ac2f8fd44472d88b 2023-03-30 16:53:54,703 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/cd1373f64fb94637ac2f8fd44472d88b, entries=150, sequenceid=262, filesize=11.8 K 2023-03-30 16:53:54,705 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 4600ee2f6bff5cc4f7b8bf671d05894d in 988ms, sequenceid=262, compaction requested=true 2023-03-30 16:53:54,705 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:54,705 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:54,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:54,706 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:54,706 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,706 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:54,707 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:54,709 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,709 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:54,709 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:54,709 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:54,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:54,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fa522a3[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:54,711 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:54,711 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:54,713 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:54,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:54,714 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:54,714 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:54,714 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:54,714 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,714 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:54,715 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:54,716 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:54,716 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:54,716 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:54,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:54,719 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:54,719 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:54,719 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:54,719 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:54,719 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:54,721 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:54,721 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:54,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:54,721 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:54,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:54,723 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:54,724 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:54,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:54,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,728 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,728 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:54,728 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:54,728 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:54,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:54,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:54,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:54,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:54,728 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:54,728 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:54,728 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,728 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:54,728 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:54,728 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:54,728 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:54,728 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:54,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:54,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:54,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:54,737 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=154.31 KB heapSize=439.92 KB 2023-03-30 16:53:54,737 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:54,737 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-30 16:53:54,737 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:54,737 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-30 16:53:54,737 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:54,737 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-30 16:53:54,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:54,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195294750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:54,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41792 deadline: 1680195294751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,808 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/19758b1e67d647f7b280126ff49d9e4a 2023-03-30 16:53:54,825 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/65fb2d3a9455400987c0899ea2bd6bd0 2023-03-30 16:53:54,828 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:54,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:54,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:54,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:54,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:54,836 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:54,836 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:54,836 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:54,836 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:54,837 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:54,837 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:54,839 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:54,839 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:54,840 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/1d377189fb434958aae987491313e42c 2023-03-30 16:53:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:54,840 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,840 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:54,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:54,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:54,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:54,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:54,842 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:54,842 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:54,842 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:54,842 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:54,842 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:54,842 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:54,842 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:54,842 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:54,842 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:54,842 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:54,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:54,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:54,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:54,844 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:54,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:54,845 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:54,845 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:54,845 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:54,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/19758b1e67d647f7b280126ff49d9e4a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/19758b1e67d647f7b280126ff49d9e4a 2023-03-30 16:53:54,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:54,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42cf231[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:54,846 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:54,847 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:54,854 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/19758b1e67d647f7b280126ff49d9e4a, entries=250, sequenceid=318, filesize=16.7 K 2023-03-30 16:53:54,856 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:54,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:54,856 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:54,856 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:54,856 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:54,856 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,857 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:54,858 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:54,858 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:54,858 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:54,858 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:54,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:54,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195294856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:54,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41792 deadline: 1680195294857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:54,861 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/65fb2d3a9455400987c0899ea2bd6bd0 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/65fb2d3a9455400987c0899ea2bd6bd0 2023-03-30 16:53:54,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:54,863 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:54,863 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:54,863 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:54,863 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:54,863 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:54,868 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:54,868 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:54,868 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:54,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:54,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:54,869 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/65fb2d3a9455400987c0899ea2bd6bd0, entries=150, sequenceid=318, filesize=11.9 K 2023-03-30 16:53:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:54,870 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:54,871 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/1d377189fb434958aae987491313e42c as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/1d377189fb434958aae987491313e42c 2023-03-30 16:53:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,874 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:54,879 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/1d377189fb434958aae987491313e42c, entries=150, sequenceid=318, filesize=11.9 K 2023-03-30 16:53:54,881 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~439.88 KB/450432, currentSize=26.84 KB/27480 for 4600ee2f6bff5cc4f7b8bf671d05894d in 144ms, sequenceid=318, compaction requested=true 2023-03-30 16:53:54,881 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:54,881 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:53:54,881 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:53:54,881 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:53:54,881 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:53:54,881 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:53:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:54,885 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53736 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:53:54,885 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48856 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:53:54,885 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/A is initiating minor compaction (all files) 2023-03-30 16:53:54,885 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/B is initiating minor compaction (all files) 2023-03-30 16:53:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,886 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/A in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:54,886 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/9c61a835c1cf4827b9a17db31ba5534d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/9a991396fbf24827a335f893269eab57, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/89c391536c5c4295bcd4a7aad2bc95e1, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/19758b1e67d647f7b280126ff49d9e4a] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=52.5 K 2023-03-30 16:53:54,886 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/B in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:54,887 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/084e3dad8cb545738861f2825de6deb0, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/ba9c6b4839e04ea0a87814c455351f58, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/6832861d49be49e09090bb75cf800d4c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/65fb2d3a9455400987c0899ea2bd6bd0] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=47.7 K 2023-03-30 16:53:54,888 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:54,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:54,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:54,892 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:54,892 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 9c61a835c1cf4827b9a17db31ba5534d, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680195231734 2023-03-30 16:53:54,891 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,892 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:54,892 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:54,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:54,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:54,892 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:54,892 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 084e3dad8cb545738861f2825de6deb0, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680195231734 2023-03-30 16:53:54,892 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:54,892 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 9a991396fbf24827a335f893269eab57, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1680195231847 2023-03-30 16:53:54,892 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:54,893 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,893 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:54,893 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:54,893 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:54,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:54,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:54,893 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 89c391536c5c4295bcd4a7aad2bc95e1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1680195232410 2023-03-30 16:53:54,893 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting ba9c6b4839e04ea0a87814c455351f58, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1680195231847 2023-03-30 16:53:54,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:54,894 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 19758b1e67d647f7b280126ff49d9e4a, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680195234105 2023-03-30 16:53:54,894 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 6832861d49be49e09090bb75cf800d4c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1680195232410 2023-03-30 16:53:54,895 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 65fb2d3a9455400987c0899ea2bd6bd0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680195234105 2023-03-30 16:53:54,922 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#A#compaction#58 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:53:54,922 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#B#compaction#57 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:53:54,956 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/6fc28b6fa6ff4040a3d3466ceab0859b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/6fc28b6fa6ff4040a3d3466ceab0859b 2023-03-30 16:53:54,963 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/B of 4600ee2f6bff5cc4f7b8bf671d05894d into 6fc28b6fa6ff4040a3d3466ceab0859b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:53:54,963 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:54,963 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/B, priority=12, startTime=1680195234881; duration=0sec 2023-03-30 16:53:54,963 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:53:54,963 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:53:54,965 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48856 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:53:54,965 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/C is initiating minor compaction (all files) 2023-03-30 16:53:54,965 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/C in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:54,965 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/635d157ae0394d94911bf0a13135ce84, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/a0809f3e316e401e99c3de62e3d6cd69, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/cd1373f64fb94637ac2f8fd44472d88b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/1d377189fb434958aae987491313e42c] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=47.7 K 2023-03-30 16:53:54,966 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 635d157ae0394d94911bf0a13135ce84, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680195231734 2023-03-30 16:53:54,966 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting a0809f3e316e401e99c3de62e3d6cd69, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1680195231847 2023-03-30 16:53:54,967 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting cd1373f64fb94637ac2f8fd44472d88b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1680195232410 2023-03-30 16:53:54,967 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 1d377189fb434958aae987491313e42c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680195234105 2023-03-30 16:53:54,978 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#C#compaction#59 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:53:54,992 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:54,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:54,994 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/e65aa8f539f549e5b0a3587ae9f2f706 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/e65aa8f539f549e5b0a3587ae9f2f706 2023-03-30 16:53:54,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:54,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:54,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:54,996 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:54,996 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:54,996 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:54,997 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:54,997 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:54,997 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:54,999 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:54,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:54,999 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:54,999 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:55,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:55,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:54,999 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:55,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:55,001 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:55,001 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:55,001 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:55,001 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:55,001 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:55,001 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:55,001 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:55,002 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-30 16:53:55,002 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/C of 4600ee2f6bff5cc4f7b8bf671d05894d into e65aa8f539f549e5b0a3587ae9f2f706(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:53:55,002 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:55,002 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/C, priority=12, startTime=1680195234881; duration=0sec 2023-03-30 16:53:55,002 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:53:55,002 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:55,003 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:55,003 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:55,003 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:55,004 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:55,004 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:55,013 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/e403e416f45846458ad9f55ef0b994ac 2023-03-30 16:53:55,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:55,065 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:55,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195235087, max timestamp=1680195235096Num uniques 859; 2023-03-30 16:53:55,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:53:55,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195235087, max timestamp=1680195235096Num uniques 859; 2023-03-30 16:53:55,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195235087, max timestamp=1680195235094Num uniques 859; 2023-03-30 16:53:55,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:53:55,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:53:55,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680195235132, max timestamp=1680195235153Num uniques 1018; 2023-03-30 16:53:55,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-03-30 16:53:55,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680195235132, max timestamp=1680195235153Num uniques 1018; 2023-03-30 16:53:55,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680195235132, max timestamp=1680195235153Num uniques 1018; 2023-03-30 16:53:55,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-03-30 16:53:55,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-03-30 16:53:55,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680195235171, max timestamp=1680195235181Num uniques 1177; 2023-03-30 16:53:55,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680195235171, max timestamp=1680195235181Num uniques 1177; 2023-03-30 16:53:55,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=12682, suffix heap size=150376,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=277 2023-03-30 16:53:55,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=12682, suffix heap size=150376,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=277 2023-03-30 16:53:55,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680195235174, max timestamp=1680195235181Num uniques 1177; 2023-03-30 16:53:55,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=8102, suffix heap size=150376,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=177 2023-03-30 16:53:55,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680195235194, max timestamp=1680195235212Num uniques 1136; 2023-03-30 16:53:55,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=15386, suffix heap size=146112,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=336 2023-03-30 16:53:55,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680195235194, max timestamp=1680195235212Num uniques 1136; 2023-03-30 16:53:55,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1680195235194, max timestamp=1680195235212Num uniques 1036; 2023-03-30 16:53:55,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=15386, suffix heap size=146112,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=336 2023-03-30 16:53:55,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=15386, suffix heap size=135712,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=336 2023-03-30 16:53:55,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:55,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195295238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1680195235204, max timestamp=1680195235237Num uniques 1195; 2023-03-30 16:53:55,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1680195235204, max timestamp=1680195235237Num uniques 1195; 2023-03-30 16:53:55,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=18090, suffix heap size=152248,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=395 2023-03-30 16:53:55,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=18090, suffix heap size=152248,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=395 2023-03-30 16:53:55,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1680195235204, max timestamp=1680195235237Num uniques 1195; 2023-03-30 16:53:55,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=18090, suffix heap size=152248,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=395 2023-03-30 16:53:55,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:55,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41792 deadline: 1680195295290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:55,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195295341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,343 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/a4de89ceb9c54479aed074961bd90d4d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/a4de89ceb9c54479aed074961bd90d4d 2023-03-30 16:53:55,349 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/A of 4600ee2f6bff5cc4f7b8bf671d05894d into a4de89ceb9c54479aed074961bd90d4d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:53:55,350 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:55,350 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/A, priority=12, startTime=1680195234881; duration=0sec 2023-03-30 16:53:55,350 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:53:55,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:55,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41792 deadline: 1680195295395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,447 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/e4f4223f3d4e49cdbf36df811c4bb0e2 2023-03-30 16:53:55,458 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/fc7f0739b19043839d979125844b578d 2023-03-30 16:53:55,467 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/e403e416f45846458ad9f55ef0b994ac as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/e403e416f45846458ad9f55ef0b994ac 2023-03-30 16:53:55,477 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/e403e416f45846458ad9f55ef0b994ac, entries=150, sequenceid=327, filesize=11.9 K 2023-03-30 16:53:55,478 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/e4f4223f3d4e49cdbf36df811c4bb0e2 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/e4f4223f3d4e49cdbf36df811c4bb0e2 2023-03-30 16:53:55,486 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/e4f4223f3d4e49cdbf36df811c4bb0e2, entries=150, sequenceid=327, filesize=11.9 K 2023-03-30 16:53:55,488 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/fc7f0739b19043839d979125844b578d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/fc7f0739b19043839d979125844b578d 2023-03-30 16:53:55,501 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/fc7f0739b19043839d979125844b578d, entries=150, sequenceid=327, filesize=11.9 K 2023-03-30 16:53:55,502 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for 4600ee2f6bff5cc4f7b8bf671d05894d in 500ms, sequenceid=327, compaction requested=false 2023-03-30 16:53:55,503 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:55,503 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:55,503 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:55,503 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:55,503 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,503 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:55,503 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:55,504 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,504 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:55,505 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,505 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:55,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:55,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e746115[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:55,506 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:55,506 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,512 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,512 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:55,512 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:55,512 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:55,513 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,513 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:55,514 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:55,514 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:55,514 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:55,514 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:55,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:55,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:55,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:55,518 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:55,518 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:55,518 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:55,518 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:55,518 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:55,521 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,521 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:55,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,521 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:55,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:55,522 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,526 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,526 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:55,526 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:55,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:55,526 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:55,526 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:55,526 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:55,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:55,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:55,526 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,526 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,527 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,527 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:55,527 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:55,527 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:55,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:55,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:55,547 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=167.72 KB heapSize=462.47 KB 2023-03-30 16:53:55,547 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:55,547 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:53:55,547 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:55,547 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:53:55,547 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:55,547 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:53:55,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:55,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195295559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:55,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41792 deadline: 1680195295599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,627 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:55,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:55,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:55,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:55,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:55,656 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:55,656 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:55,656 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:55,656 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:55,656 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,657 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:55,658 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,658 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:55,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:55,658 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,658 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:55,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:55,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:55,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:55,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:55,665 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:55,665 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:55,665 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:55,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:55,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195295665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,669 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:55,669 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:55,670 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:55,670 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:55,670 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:55,670 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:55,670 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:55,674 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:55,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:55,674 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:55,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:55,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:55,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d904942[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:55,680 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:55,680 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,683 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,683 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:55,683 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:55,683 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:55,690 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,691 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:55,691 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:55,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:55,697 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:55,697 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:55,697 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:55,697 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:55,697 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:55,700 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,700 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:55,700 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:55,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:55,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:55,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,702 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:55,704 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,708 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:55,708 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:55,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:55,708 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:55,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:55,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:55,709 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:55,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:55,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:55,709 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:55,710 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,710 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:55,710 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:55,710 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,710 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,711 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,711 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,809 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:55,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:55,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:55,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:55,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:55,813 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:55,813 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:55,813 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:55,813 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:55,814 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,814 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:55,815 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:55,815 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:55,816 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,816 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:55,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:55,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:55,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:55,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:55,817 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:55,817 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:55,817 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:55,817 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:55,818 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:55,818 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:55,818 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:55,818 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:55,818 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:55,818 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:55,818 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:55,818 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,818 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:55,818 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:55,820 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,820 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:55,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:55,821 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,821 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:55,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:55,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bc6d069[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:55,823 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:55,823 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,824 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,824 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:55,824 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:55,824 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:55,825 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,825 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:55,832 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,832 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:55,832 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:55,832 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:55,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:55,834 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:55,834 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:55,834 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:55,834 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:55,834 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:55,836 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,836 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:55,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,836 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:55,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:55,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:55,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:55,837 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,838 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,840 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:55,840 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,840 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:55,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:55,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:55,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:55,840 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:55,840 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:55,841 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:55,841 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:55,841 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:55,841 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,841 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:55,841 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,841 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:55,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:55,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195295874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:55,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41792 deadline: 1680195295901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,941 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:55,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:55,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:55,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:55,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:55,945 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:55,945 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:55,945 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:55,945 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:55,946 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,946 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:55,949 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,949 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,949 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:55,954 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:55,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:55,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:55,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:55,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:55,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:55,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:55,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:55,960 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:55,960 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:55,960 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:55,960 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:55,960 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:55,961 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:55,961 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:55,961 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/294d94830cdb49a3afd6151522803695 2023-03-30 16:53:55,961 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:55,961 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:55,961 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:55,961 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:55,961 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,961 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:55,961 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:55,964 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,964 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:55,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:55,965 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,965 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:55,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:55,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:55,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1818185a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:55,969 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:55,970 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,974 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,974 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:55,974 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:55,974 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:55,974 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,974 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:55,976 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:55,976 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:55,976 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:55,976 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:55,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:55,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:55,981 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:55,981 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:55,981 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:55,982 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:55,982 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:55,986 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,986 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,986 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:55,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:55,987 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,988 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,991 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:55,991 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:55,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:55,991 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,991 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:55,991 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:55,992 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:55,992 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:55,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:55,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:55,992 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,992 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:55,992 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:55,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:55,992 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:55,992 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:56,092 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:56,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:56,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:56,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:56,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:56,097 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:56,097 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:56,097 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:56,097 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:56,098 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,098 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:56,099 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,099 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:56,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:56,100 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,100 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:56,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:56,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:56,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:56,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:56,101 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:56,101 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:56,101 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:56,101 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:56,101 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:56,101 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:56,101 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:56,101 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:56,101 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:56,102 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:56,102 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:56,102 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,102 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:56,102 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:56,103 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,103 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:56,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:56,104 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:56,104 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:56,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:56,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:56,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:56,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:56,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3549ae85[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:56,106 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:56,106 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,108 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,108 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:56,108 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:56,108 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:56,108 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,108 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:56,110 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:56,110 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:56,110 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:56,110 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:56,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:56,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:56,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:56,113 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:56,113 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:56,113 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:56,113 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:56,113 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:56,115 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,115 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:56,115 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:56,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:56,116 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:56,117 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,120 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,120 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:56,120 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:56,120 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,120 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,120 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:56,120 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,120 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:56,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:56,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:56,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:56,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:56,120 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:56,120 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,120 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:56,121 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:56,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:56,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:56,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:56,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195296180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,221 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:56,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:56,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:56,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:56,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:56,225 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:56,225 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:56,225 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:56,225 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:56,225 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,226 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:56,229 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:56,229 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:56,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:56,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:56,230 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,230 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:56,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:56,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:56,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:56,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:56,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:56,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:56,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:56,245 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:56,245 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:56,245 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:56,245 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:56,245 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:56,245 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:56,245 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:56,245 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:56,245 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,245 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:56,245 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:56,247 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,248 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,248 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:56,248 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:56,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:56,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:56,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a8c2b37[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:56,252 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:56,252 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,255 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,256 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,256 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,256 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:56,256 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:56,256 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:56,259 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:56,259 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:56,259 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:56,259 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:56,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:56,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:56,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:56,263 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:56,263 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:56,263 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:56,263 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:56,263 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:56,270 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,270 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:56,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,271 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:56,271 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,273 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,278 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:56,278 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,278 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:56,278 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:56,278 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:56,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:56,278 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:56,278 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,278 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,278 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:56,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:56,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:56,278 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:56,279 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:56,279 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:56,379 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:56,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:56,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:56,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:56,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:56,382 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:56,382 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:56,382 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:56,382 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:56,382 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,382 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:56,384 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:56,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:56,384 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:56,385 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,385 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:56,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:56,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:56,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:56,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:56,386 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:56,386 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:56,386 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:56,387 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:56,387 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:56,387 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:56,388 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:56,388 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:56,388 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:56,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:56,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:56,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:56,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:56,389 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,389 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:56,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:56,390 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,390 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:56,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:56,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:56,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:56,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:56,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:56,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6784f397[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:56,392 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:56,392 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,394 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/02ca5c017c784549b72b5d9d5bc4ff35 2023-03-30 16:53:56,394 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,394 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:56,394 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:56,394 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:56,395 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,395 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:56,396 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:56,396 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,396 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:56,396 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:56,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:56,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:56,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:56,404 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:56,404 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:56,404 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:56,404 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:56,404 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:56,406 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,406 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:56,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,406 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:56,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:56,408 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:56,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:56,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41792 deadline: 1680195296405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,409 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:56,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,411 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,411 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:56,411 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:56,411 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,411 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:56,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:56,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:56,411 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,411 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:56,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:56,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:56,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:56,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:56,412 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:56,412 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:56,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:56,416 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/afa6aa966b9a45b0ad74abfe25d94045 2023-03-30 16:53:56,421 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/294d94830cdb49a3afd6151522803695 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/294d94830cdb49a3afd6151522803695 2023-03-30 16:53:56,426 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/294d94830cdb49a3afd6151522803695, entries=200, sequenceid=434, filesize=14.3 K 2023-03-30 16:53:56,427 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/02ca5c017c784549b72b5d9d5bc4ff35 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/02ca5c017c784549b72b5d9d5bc4ff35 2023-03-30 16:53:56,433 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/02ca5c017c784549b72b5d9d5bc4ff35, entries=150, sequenceid=434, filesize=11.9 K 2023-03-30 16:53:56,435 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/afa6aa966b9a45b0ad74abfe25d94045 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/afa6aa966b9a45b0ad74abfe25d94045 2023-03-30 16:53:56,439 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/afa6aa966b9a45b0ad74abfe25d94045, entries=150, sequenceid=434, filesize=11.9 K 2023-03-30 16:53:56,440 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~462.42 KB/473520, currentSize=20.13 KB/20610 for 4600ee2f6bff5cc4f7b8bf671d05894d in 893ms, sequenceid=434, compaction requested=true 2023-03-30 16:53:56,440 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:56,440 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:53:56,441 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:53:56,441 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:53:56,441 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-30 16:53:56,441 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:53:56,443 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39646 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:53:56,443 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:53:56,443 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/A is initiating minor compaction (all files) 2023-03-30 16:53:56,443 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/B is initiating minor compaction (all files) 2023-03-30 16:53:56,443 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/A in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:56,443 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/B in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:56,443 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/a4de89ceb9c54479aed074961bd90d4d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/e403e416f45846458ad9f55ef0b994ac, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/294d94830cdb49a3afd6151522803695] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=38.7 K 2023-03-30 16:53:56,443 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/6fc28b6fa6ff4040a3d3466ceab0859b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/e4f4223f3d4e49cdbf36df811c4bb0e2, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/02ca5c017c784549b72b5d9d5bc4ff35] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=36.3 K 2023-03-30 16:53:56,444 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting a4de89ceb9c54479aed074961bd90d4d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680195234105 2023-03-30 16:53:56,444 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 6fc28b6fa6ff4040a3d3466ceab0859b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680195234105 2023-03-30 16:53:56,444 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting e403e416f45846458ad9f55ef0b994ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680195234742 2023-03-30 16:53:56,444 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting e4f4223f3d4e49cdbf36df811c4bb0e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680195234742 2023-03-30 16:53:56,445 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 294d94830cdb49a3afd6151522803695, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1680195235275 2023-03-30 16:53:56,445 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 02ca5c017c784549b72b5d9d5bc4ff35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1680195235275 2023-03-30 16:53:56,455 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#B#compaction#66 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:53:56,459 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#A#compaction#67 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:53:56,512 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:56,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:56,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:56,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:56,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:56,516 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:56,516 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:56,516 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:56,516 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:56,516 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,516 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:56,520 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:56,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:56,520 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:56,520 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,520 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:56,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:56,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:56,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:56,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:56,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:56,522 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:56,522 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:56,522 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-30 16:53:56,522 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:56,523 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:56,523 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:56,523 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:56,523 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:56,523 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:56,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:56,582 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:56,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680195236592, max timestamp=1680195236605Num uniques 859; 2023-03-30 16:53:56,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:53:56,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680195236592, max timestamp=1680195236605Num uniques 859; 2023-03-30 16:53:56,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:53:56,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680195236592, max timestamp=1680195236605Num uniques 859; 2023-03-30 16:53:56,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-30 16:53:56,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680195236615, max timestamp=1680195236635Num uniques 1068; 2023-03-30 16:53:56,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680195236615, max timestamp=1680195236635Num uniques 1009; 2023-03-30 16:53:56,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=12268, suffix heap size=132904,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=268 2023-03-30 16:53:56,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-30 16:53:56,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680195236615, max timestamp=1680195236635Num uniques 1068; 2023-03-30 16:53:56,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-30 16:53:56,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680195236659, max timestamp=1680195236666Num uniques 1127; 2023-03-30 16:53:56,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-03-30 16:53:56,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680195236659, max timestamp=1680195236666Num uniques 1127; 2023-03-30 16:53:56,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-03-30 16:53:56,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680195236659, max timestamp=1680195236666Num uniques 1127; 2023-03-30 16:53:56,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-03-30 16:53:56,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1680195236689, max timestamp=1680195236700Num uniques 1036; 2023-03-30 16:53:56,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=17676, suffix heap size=135712,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=386 2023-03-30 16:53:56,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1680195236689, max timestamp=1680195236700Num uniques 1036; 2023-03-30 16:53:56,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=17676, suffix heap size=135712,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=386 2023-03-30 16:53:56,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1680195236689, max timestamp=1680195236700Num uniques 1036; 2023-03-30 16:53:56,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=17676, suffix heap size=135712,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=386 2023-03-30 16:53:56,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1680195236714, max timestamp=1680195236727Num uniques 1245; 2023-03-30 16:53:56,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=57020, new segment data size=18090, suffix heap size=157448,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1245, new segment cells count=395 2023-03-30 16:53:56,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1680195236714, max timestamp=1680195236727Num uniques 1245; 2023-03-30 16:53:56,731 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=57020, new segment data size=18090, suffix heap size=157448,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1245, new segment cells count=395 2023-03-30 16:53:56,732 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1680195236714, max timestamp=1680195236727Num uniques 1245; 2023-03-30 16:53:56,732 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=57020, new segment data size=18090, suffix heap size=157448,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1245, new segment cells count=395 2023-03-30 16:53:56,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:56,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41798 deadline: 1680195296770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:56,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41828 deadline: 1680195296770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:56,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195296770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:56,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41814 deadline: 1680195296773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,883 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/208f483511624e1e8495a1c43d70442f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/208f483511624e1e8495a1c43d70442f 2023-03-30 16:53:56,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:56,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41814 deadline: 1680195296881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:56,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41798 deadline: 1680195296881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:56,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:56,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41828 deadline: 1680195296881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,886 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/90842eea8b334197ae3bb21a0601c555 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/90842eea8b334197ae3bb21a0601c555 2023-03-30 16:53:56,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195296881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:56,895 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/A of 4600ee2f6bff5cc4f7b8bf671d05894d into 208f483511624e1e8495a1c43d70442f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:53:56,895 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:56,895 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/A, priority=13, startTime=1680195236440; duration=0sec 2023-03-30 16:53:56,895 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:53:56,895 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:53:56,897 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:53:56,897 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/C is initiating minor compaction (all files) 2023-03-30 16:53:56,898 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/C in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:56,898 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/e65aa8f539f549e5b0a3587ae9f2f706, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/fc7f0739b19043839d979125844b578d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/afa6aa966b9a45b0ad74abfe25d94045] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=36.3 K 2023-03-30 16:53:56,898 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting e65aa8f539f549e5b0a3587ae9f2f706, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680195234105 2023-03-30 16:53:56,899 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting fc7f0739b19043839d979125844b578d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680195234742 2023-03-30 16:53:56,899 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting afa6aa966b9a45b0ad74abfe25d94045, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1680195235275 2023-03-30 16:53:56,904 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/B of 4600ee2f6bff5cc4f7b8bf671d05894d into 90842eea8b334197ae3bb21a0601c555(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:53:56,904 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:56,904 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/B, priority=13, startTime=1680195236441; duration=0sec 2023-03-30 16:53:56,904 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:53:56,908 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#C#compaction#69 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:53:56,945 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/f6f6ff705bbb4cb392b29549690d2aa5 2023-03-30 16:53:56,969 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/0880d9952c3f4cd69c571c3bc0d419f2 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/0880d9952c3f4cd69c571c3bc0d419f2 2023-03-30 16:53:56,977 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/79d11945b3ea47b88f8cb6a8eedcb8f9 2023-03-30 16:53:56,982 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/C of 4600ee2f6bff5cc4f7b8bf671d05894d into 0880d9952c3f4cd69c571c3bc0d419f2(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:53:56,982 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:56,982 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/C, priority=13, startTime=1680195236441; duration=0sec 2023-03-30 16:53:56,982 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:53:56,998 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/11f465ff1a724206bbc02b07d92876bb 2023-03-30 16:53:57,004 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/f6f6ff705bbb4cb392b29549690d2aa5 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/f6f6ff705bbb4cb392b29549690d2aa5 2023-03-30 16:53:57,009 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/f6f6ff705bbb4cb392b29549690d2aa5, entries=100, sequenceid=440, filesize=9.5 K 2023-03-30 16:53:57,011 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/79d11945b3ea47b88f8cb6a8eedcb8f9 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/79d11945b3ea47b88f8cb6a8eedcb8f9 2023-03-30 16:53:57,016 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/79d11945b3ea47b88f8cb6a8eedcb8f9, entries=100, sequenceid=440, filesize=9.5 K 2023-03-30 16:53:57,017 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/11f465ff1a724206bbc02b07d92876bb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/11f465ff1a724206bbc02b07d92876bb 2023-03-30 16:53:57,025 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/11f465ff1a724206bbc02b07d92876bb, entries=100, sequenceid=440, filesize=9.5 K 2023-03-30 16:53:57,026 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=167.72 KB/171750 for 4600ee2f6bff5cc4f7b8bf671d05894d in 504ms, sequenceid=440, compaction requested=false 2023-03-30 16:53:57,026 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:57,026 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:57,026 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:57,026 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:57,026 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,026 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:57,026 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:57,028 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,028 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:57,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:57,029 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:57,029 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:57,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:57,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:57,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:57,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f86edf5[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:57,031 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:57,031 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,032 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,032 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:57,032 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:57,032 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:57,033 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,033 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:57,040 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:57,040 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:57,040 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:57,041 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:57,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:57,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:57,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:57,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:57,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:57,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:57,043 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:57,044 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:57,044 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:57,044 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:57,044 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:57,049 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,049 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,049 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:57,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:57,050 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,054 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:57,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,057 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:57,057 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,057 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:57,057 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:57,057 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,057 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,057 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:57,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:57,057 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:57,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:57,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:57,057 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,058 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:57,058 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:57,058 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:57,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:57,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:57,089 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=174.43 KB heapSize=483.09 KB 2023-03-30 16:53:57,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680195236738, max timestamp=1680195237087Num uniques 1254; 2023-03-30 16:53:57,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=57430, new segment data size=7050, suffix heap size=158384,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1254, new segment cells count=154 2023-03-30 16:53:57,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680195236738, max timestamp=1680195237087Num uniques 1254; 2023-03-30 16:53:57,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=57430, new segment data size=7050, suffix heap size=158384,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1254, new segment cells count=154 2023-03-30 16:53:57,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680195236738, max timestamp=1680195237087Num uniques 1254; 2023-03-30 16:53:57,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=57430, new segment data size=7050, suffix heap size=158384,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1254, new segment cells count=154 2023-03-30 16:53:57,091 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:57,091 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:53:57,091 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:57,091 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:53:57,091 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:57,091 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:53:57,104 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/9566ec4c0e7c4de0a69ca143efb4bfaf 2023-03-30 16:53:57,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680195237101, max timestamp=1680195237111Num uniques 859; 2023-03-30 16:53:57,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 16:53:57,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680195237101, max timestamp=1680195237111Num uniques 859; 2023-03-30 16:53:57,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680195237101, max timestamp=1680195237111Num uniques 859; 2023-03-30 16:53:57,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 16:53:57,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 16:53:57,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41828 deadline: 1680195297133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680195237125, max timestamp=1680195237136Num uniques 1118; 2023-03-30 16:53:57,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-03-30 16:53:57,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680195237125, max timestamp=1680195237136Num uniques 1118; 2023-03-30 16:53:57,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680195237125, max timestamp=1680195237136Num uniques 1118; 2023-03-30 16:53:57,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-03-30 16:53:57,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-03-30 16:53:57,158 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:57,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:57,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:57,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:57,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:57,162 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:57,162 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:57,162 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:57,162 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:57,163 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,163 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:57,164 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:57,164 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:57,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:57,164 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,164 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:57,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:57,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195297161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:57,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:57,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:57,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:57,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:57,166 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:57,166 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:57,166 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:57,166 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:57,166 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:57,166 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:57,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1680195237153, max timestamp=1680195237164Num uniques 1127; 2023-03-30 16:53:57,166 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:57,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-03-30 16:53:57,166 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:57,166 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:57,167 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:57,167 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:57,167 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1680195237153, max timestamp=1680195237164Num uniques 1127; 2023-03-30 16:53:57,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-03-30 16:53:57,167 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:57,167 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:57,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1680195237153, max timestamp=1680195237164Num uniques 1127; 2023-03-30 16:53:57,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-03-30 16:53:57,168 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,168 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:57,169 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,169 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:57,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:57,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47c34106[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:57,171 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:57,186 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,189 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,190 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,190 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:57,190 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:57,190 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:57,190 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:57,192 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:57,192 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:57,192 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:57,192 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:57,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:57,197 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:57,197 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:57,197 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:57,197 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:57,197 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:57,201 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,201 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:57,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,201 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:57,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:57,202 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,203 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,209 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,209 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:57,209 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:57,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:57,209 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:57,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:57,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:57,209 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:57,209 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:57,209 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:57,210 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:57,210 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,210 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:57,210 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,210 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:57,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:57,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680195237218, max timestamp=1680195237259Num uniques 1186; 2023-03-30 16:53:57,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=13096, suffix heap size=151312,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=286 2023-03-30 16:53:57,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680195237218, max timestamp=1680195237259Num uniques 1186; 2023-03-30 16:53:57,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680195237218, max timestamp=1680195237259Num uniques 1186; 2023-03-30 16:53:57,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=13096, suffix heap size=151312,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=286 2023-03-30 16:53:57,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=13096, suffix heap size=151312,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=286 2023-03-30 16:53:57,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195297279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41828 deadline: 1680195297279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680195237272, max timestamp=1680195237281Num uniques 1145; 2023-03-30 16:53:57,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680195237272, max timestamp=1680195237281Num uniques 1145; 2023-03-30 16:53:57,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-03-30 16:53:57,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-03-30 16:53:57,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680195237272, max timestamp=1680195237281Num uniques 1145; 2023-03-30 16:53:57,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-03-30 16:53:57,310 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:57,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:57,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:57,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:57,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:57,313 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:57,313 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:57,313 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:57,313 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:57,314 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,314 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:57,316 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:57,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:57,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,317 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:57,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:57,319 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,319 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:57,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:57,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:57,325 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:57,326 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:57,326 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:57,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680195237315, max timestamp=1680195237325Num uniques 1104; 2023-03-30 16:53:57,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-30 16:53:57,328 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:57,328 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:57,328 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:57,328 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:57,328 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:57,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680195237315, max timestamp=1680195237325Num uniques 1104; 2023-03-30 16:53:57,328 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:57,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-30 16:53:57,328 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:57,329 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:57,329 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,329 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:57,329 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:57,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680195237315, max timestamp=1680195237325Num uniques 1104; 2023-03-30 16:53:57,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-30 16:53:57,330 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,330 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:57,332 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:57,332 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:57,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:57,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c5d1f5a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:57,334 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:57,335 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,337 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,337 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:57,337 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:57,337 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:57,337 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,338 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:57,339 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:57,339 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,339 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:57,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:57,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:57,339 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:57,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:57,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:57,342 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:57,342 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:57,342 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:57,342 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:57,342 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:57,343 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,343 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:57,344 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:57,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:57,344 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,346 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:57,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,349 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,349 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:57,349 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:57,349 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,349 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,349 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:57,349 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:57,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:57,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:57,349 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:57,349 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:57,349 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,349 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:57,349 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:57,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:57,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:57,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1680195237348, max timestamp=1680195237380Num uniques 1013; 2023-03-30 16:53:57,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=14328, suffix heap size=133320,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=313 2023-03-30 16:53:57,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1680195237348, max timestamp=1680195237380Num uniques 1013; 2023-03-30 16:53:57,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1680195237348, max timestamp=1680195237380Num uniques 1013; 2023-03-30 16:53:57,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=14328, suffix heap size=133320,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=313 2023-03-30 16:53:57,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=14328, suffix heap size=133320,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=313 2023-03-30 16:53:57,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41814 deadline: 1680195297400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41798 deadline: 1680195297403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195297404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41828 deadline: 1680195297404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41792 deadline: 1680195297413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,449 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:57,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:57,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:57,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:57,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:57,453 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:57,453 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:57,453 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:57,453 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:57,454 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,454 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:57,456 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:57,456 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:57,456 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,456 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:57,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:57,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:57,457 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:57,457 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:57,457 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:57,458 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:57,458 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:57,458 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:57,458 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:57,458 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:57,458 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:57,458 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:57,458 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:57,458 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:57,458 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:57,458 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,458 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:57,458 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:57,461 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,461 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:57,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:57,461 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:57,462 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:57,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:57,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:57,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:57,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c055d0d[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:57,464 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:57,464 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,465 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,465 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:57,465 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:57,466 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:57,466 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,466 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:57,467 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:57,467 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:57,467 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:57,467 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:57,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:57,470 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:57,470 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:57,470 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:57,470 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:57,470 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:57,472 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,472 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,472 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:57,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:57,472 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,474 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,476 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,476 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:57,476 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:57,476 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:57,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:57,476 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:57,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:57,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:57,476 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:57,476 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,476 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,476 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:57,476 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:57,476 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:57,476 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:57,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:57,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41814 deadline: 1680195297505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41798 deadline: 1680195297505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195297507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41828 deadline: 1680195297507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,523 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/61d01981a16d45359ac2897fe552b8b7 2023-03-30 16:53:57,537 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/4a6baecd9d004cbcb0a52da252c2c2ac 2023-03-30 16:53:57,548 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/9566ec4c0e7c4de0a69ca143efb4bfaf as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/9566ec4c0e7c4de0a69ca143efb4bfaf 2023-03-30 16:53:57,555 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/9566ec4c0e7c4de0a69ca143efb4bfaf, entries=150, sequenceid=551, filesize=11.9 K 2023-03-30 16:53:57,556 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/61d01981a16d45359ac2897fe552b8b7 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/61d01981a16d45359ac2897fe552b8b7 2023-03-30 16:53:57,567 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/61d01981a16d45359ac2897fe552b8b7, entries=150, sequenceid=551, filesize=11.9 K 2023-03-30 16:53:57,569 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/4a6baecd9d004cbcb0a52da252c2c2ac as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/4a6baecd9d004cbcb0a52da252c2c2ac 2023-03-30 16:53:57,575 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/4a6baecd9d004cbcb0a52da252c2c2ac, entries=150, sequenceid=551, filesize=11.9 K 2023-03-30 16:53:57,576 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:57,577 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.58 KB/88656, currentSize=154.31 KB/158010 for 4600ee2f6bff5cc4f7b8bf671d05894d in 488ms, sequenceid=551, compaction requested=true 2023-03-30 16:53:57,577 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:57,577 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:53:57,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:57,577 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:53:57,577 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:53:57,577 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:53:57,577 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:53:57,579 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:53:57,579 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:53:57,579 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/A is initiating minor compaction (all files) 2023-03-30 16:53:57,579 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/B is initiating minor compaction (all files) 2023-03-30 16:53:57,579 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/A in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:57,579 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/B in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:57,579 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/208f483511624e1e8495a1c43d70442f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/f6f6ff705bbb4cb392b29549690d2aa5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/9566ec4c0e7c4de0a69ca143efb4bfaf] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=34.0 K 2023-03-30 16:53:57,579 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/90842eea8b334197ae3bb21a0601c555, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/79d11945b3ea47b88f8cb6a8eedcb8f9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/61d01981a16d45359ac2897fe552b8b7] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=34.0 K 2023-03-30 16:53:57,579 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 208f483511624e1e8495a1c43d70442f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1680195235275 2023-03-30 16:53:57,580 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 90842eea8b334197ae3bb21a0601c555, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1680195235275 2023-03-30 16:53:57,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:57,580 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting f6f6ff705bbb4cb392b29549690d2aa5, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1680195235555 2023-03-30 16:53:57,580 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 79d11945b3ea47b88f8cb6a8eedcb8f9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1680195235555 2023-03-30 16:53:57,580 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 9566ec4c0e7c4de0a69ca143efb4bfaf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680195236765 2023-03-30 16:53:57,581 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 61d01981a16d45359ac2897fe552b8b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680195236765 2023-03-30 16:53:57,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:57,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:57,584 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:57,584 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:57,584 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:57,584 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:57,585 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,585 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:57,588 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:57,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:57,588 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:57,589 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,589 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:57,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:57,590 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#A#compaction#75 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:53:57,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:57,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:57,590 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:57,591 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:57,591 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:57,591 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:57,591 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:57,591 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:57,591 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:57,591 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=154.31 KB heapSize=430.31 KB 2023-03-30 16:53:57,591 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:57,592 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#B#compaction#76 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:53:57,592 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:53:57,592 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:57,592 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:53:57,592 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:57,592 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:53:57,631 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/d18ba9fb18f846e5820b736a11c1796c 2023-03-30 16:53:57,650 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/9692fc75c84440b1a347cca8b519c8f5 2023-03-30 16:53:57,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:57,710 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:57,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195297712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41828 deadline: 1680195297711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41798 deadline: 1680195297716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41814 deadline: 1680195297717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195297818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41828 deadline: 1680195297818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41798 deadline: 1680195297820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:57,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:57,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41814 deadline: 1680195297820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,023 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/44382a4bcb684ec0879488c90a2d7ce9 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/44382a4bcb684ec0879488c90a2d7ce9 2023-03-30 16:53:58,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:58,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195298021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:58,024 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/4c12935536d848caafb14362b30b9581 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/4c12935536d848caafb14362b30b9581 2023-03-30 16:53:58,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41828 deadline: 1680195298022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:58,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41814 deadline: 1680195298027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,032 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/A of 4600ee2f6bff5cc4f7b8bf671d05894d into 44382a4bcb684ec0879488c90a2d7ce9(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:53:58,032 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:58,032 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/A, priority=13, startTime=1680195237577; duration=0sec 2023-03-30 16:53:58,032 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:53:58,032 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:53:58,033 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:53:58,034 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/C is initiating minor compaction (all files) 2023-03-30 16:53:58,034 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/B of 4600ee2f6bff5cc4f7b8bf671d05894d into 4c12935536d848caafb14362b30b9581(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:53:58,034 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/C in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:58,034 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:58,034 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/B, priority=13, startTime=1680195237577; duration=0sec 2023-03-30 16:53:58,034 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/0880d9952c3f4cd69c571c3bc0d419f2, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/11f465ff1a724206bbc02b07d92876bb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/4a6baecd9d004cbcb0a52da252c2c2ac] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=34.0 K 2023-03-30 16:53:58,034 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:53:58,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:58,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41798 deadline: 1680195298031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,034 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 0880d9952c3f4cd69c571c3bc0d419f2, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1680195235275 2023-03-30 16:53:58,036 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 11f465ff1a724206bbc02b07d92876bb, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1680195235555 2023-03-30 16:53:58,036 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 4a6baecd9d004cbcb0a52da252c2c2ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680195236765 2023-03-30 16:53:58,054 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#C#compaction#80 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:53:58,073 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/52495e1f61564c7f8d4d1993612712eb 2023-03-30 16:53:58,081 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/d18ba9fb18f846e5820b736a11c1796c as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/d18ba9fb18f846e5820b736a11c1796c 2023-03-30 16:53:58,088 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/d18ba9fb18f846e5820b736a11c1796c, entries=150, sequenceid=691, filesize=11.9 K 2023-03-30 16:53:58,089 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/9692fc75c84440b1a347cca8b519c8f5 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/9692fc75c84440b1a347cca8b519c8f5 2023-03-30 16:53:58,096 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/9692fc75c84440b1a347cca8b519c8f5, entries=150, sequenceid=691, filesize=11.9 K 2023-03-30 16:53:58,097 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/52495e1f61564c7f8d4d1993612712eb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/52495e1f61564c7f8d4d1993612712eb 2023-03-30 16:53:58,118 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/52495e1f61564c7f8d4d1993612712eb, entries=150, sequenceid=691, filesize=11.9 K 2023-03-30 16:53:58,119 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~430.27 KB/440592, currentSize=40.25 KB/41220 for 4600ee2f6bff5cc4f7b8bf671d05894d in 528ms, sequenceid=691, compaction requested=false 2023-03-30 16:53:58,119 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:58,119 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:58,119 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:58,119 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:58,119 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,120 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:58,120 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:58,121 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:58,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:58,122 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:58,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:58,122 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:58,122 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:58,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:58,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:58,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:58,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:58,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4464e83d[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:58,124 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:58,124 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:58,125 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:58,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:58,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:58,125 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:58,125 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:58,125 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:58,126 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,126 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:58,127 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:58,127 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:58,127 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:58,127 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:58,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:58,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:58,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:58,130 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:58,130 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:58,130 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:58,130 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:58,130 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:58,133 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:58,133 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:58,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:58,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:58,133 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:58,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:58,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:58,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:58,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:58,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:58,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:58,134 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:58,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:58,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:58,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:58,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,135 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:58,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:58,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,138 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,138 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:58,138 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:58,138 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:58,138 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:58,138 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:58,138 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:58,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:58,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:58,138 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:58,138 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:58,138 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:58,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:58,138 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:58,138 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:58,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:58,238 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:58,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:53:58,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:53:58,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:53:58,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:53:58,242 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:53:58,243 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:58,243 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:53:58,243 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:53:58,243 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:58,243 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:58,250 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:58,250 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:58,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:58,250 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:58,251 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:53:58,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:53:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:58,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:53:58,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:53:58,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:53:58,252 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:53:58,252 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:53:58,252 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:53:58,256 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:53:58,256 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:53:58,256 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:58,256 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. started... 2023-03-30 16:53:58,256 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-30 16:53:58,256 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:53:58,256 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:58,256 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:53:58,256 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:58,257 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:53:58,257 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:53:58,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:53:58,326 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. as already flushing 2023-03-30 16:53:58,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195238341, max timestamp=1680195238344Num uniques 859; 2023-03-30 16:53:58,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:53:58,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195238341, max timestamp=1680195238344Num uniques 859; 2023-03-30 16:53:58,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195238341, max timestamp=1680195238344Num uniques 859; 2023-03-30 16:53:58,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:53:58,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:53:58,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:58,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41814 deadline: 1680195298359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:58,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:58,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41798 deadline: 1680195298360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195298360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1680195238354, max timestamp=1680195238362Num uniques 1018; 2023-03-30 16:53:58,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1680195238354, max timestamp=1680195238362Num uniques 1018; 2023-03-30 16:53:58,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1680195238354, max timestamp=1680195238362Num uniques 1018; 2023-03-30 16:53:58,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=16848, suffix heap size=133840,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=368 2023-03-30 16:53:58,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=16848, suffix heap size=133840,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=368 2023-03-30 16:53:58,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=16848, suffix heap size=133840,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=368 2023-03-30 16:53:58,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:58,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41828 deadline: 1680195298398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:58,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195298466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:58,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41814 deadline: 1680195298466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:58,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41798 deadline: 1680195298466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,470 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/72ba2ad449a84dc29906282d89e21ca1 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/72ba2ad449a84dc29906282d89e21ca1 2023-03-30 16:53:58,484 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/C of 4600ee2f6bff5cc4f7b8bf671d05894d into 72ba2ad449a84dc29906282d89e21ca1(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:53:58,484 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:58,484 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/C, priority=13, startTime=1680195237577; duration=0sec 2023-03-30 16:53:58,484 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:53:58,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:58,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41828 deadline: 1680195298502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:58,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195298669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:58,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41814 deadline: 1680195298669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:58,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41798 deadline: 1680195298669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,689 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/e5a524d57e864563995f40ac28121df1 2023-03-30 16:53:58,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:58,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41828 deadline: 1680195298703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:58,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41798 deadline: 1680195298972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:58,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195298972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:58,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:58,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41814 deadline: 1680195298972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:59,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41828 deadline: 1680195299007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,102 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/51575202ea8a4663892d436cdd48faa6 2023-03-30 16:53:59,337 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51ef1151 to 127.0.0.1:58435 2023-03-30 16:53:59,337 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:53:59,339 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ec07124 to 127.0.0.1:58435 2023-03-30 16:53:59,339 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:53:59,343 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00375c9d to 127.0.0.1:58435 2023-03-30 16:53:59,343 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:53:59,345 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ec2fae8 to 127.0.0.1:58435 2023-03-30 16:53:59,345 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:53:59,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:59,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41792 deadline: 1680195299432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,434 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:53:59,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:59,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41814 deadline: 1680195299476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:59,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41762 deadline: 1680195299476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:59,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41798 deadline: 1680195299478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,514 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/4bd0ea2a5ccb4d5287df545cd1ca19a9 2023-03-30 16:53:59,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:53:59,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41828 deadline: 1680195299515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,519 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/e5a524d57e864563995f40ac28121df1 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/e5a524d57e864563995f40ac28121df1 2023-03-30 16:53:59,523 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/e5a524d57e864563995f40ac28121df1, entries=150, sequenceid=702, filesize=11.9 K 2023-03-30 16:53:59,524 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/51575202ea8a4663892d436cdd48faa6 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/51575202ea8a4663892d436cdd48faa6 2023-03-30 16:53:59,528 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/51575202ea8a4663892d436cdd48faa6, entries=150, sequenceid=702, filesize=11.9 K 2023-03-30 16:53:59,529 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/4bd0ea2a5ccb4d5287df545cd1ca19a9 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/4bd0ea2a5ccb4d5287df545cd1ca19a9 2023-03-30 16:53:59,534 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/4bd0ea2a5ccb4d5287df545cd1ca19a9, entries=150, sequenceid=702, filesize=11.9 K 2023-03-30 16:53:59,534 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 4600ee2f6bff5cc4f7b8bf671d05894d in 1278ms, sequenceid=702, compaction requested=true 2023-03-30 16:53:59,534 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:53:59,534 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:53:59,534 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:53:59,534 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:53:59,535 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,535 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:53:59,535 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:53:59,536 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:59,536 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:53:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:59,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:53:59,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19725ac6[Count = 0] remaining members to acquire global barrier 2023-03-30 16:53:59,538 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:53:59,538 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:59,539 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:59,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:59,539 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:53:59,539 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:53:59,540 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:53:59,540 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,540 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:53:59,544 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:53:59,544 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:59,544 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,544 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:53:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:53:59,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:53:59,546 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:53:59,546 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:53:59,546 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:53:59,546 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:53:59,546 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:53:59,547 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:59,547 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:59,547 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:53:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:53:59,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:53:59,548 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:53:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:59,549 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:53:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:53:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,551 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,551 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:53:59,551 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:59,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:53:59,551 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:53:59,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:53:59,551 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:53:59,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:53:59,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:53:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:53:59,551 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:53:59,551 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:53:59,551 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:59,551 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:53:59,551 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:53:59,551 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:53:59,552 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:53:59,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:53:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:53:59,652 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:53:59,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:00,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:54:00,479 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=154.31 KB heapSize=425.16 KB 2023-03-30 16:54:00,479 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:54:00,479 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12adc6a1 to 127.0.0.1:58435 2023-03-30 16:54:00,479 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:00,479 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:00,479 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:54:00,479 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:00,479 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:54:00,479 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:00,484 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7b52aa42 to 127.0.0.1:58435 2023-03-30 16:54:00,484 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:00,486 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2732b8c5 to 127.0.0.1:58435 2023-03-30 16:54:00,486 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:00,487 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=756 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/51b543c860db4264bf71972ba0283eae 2023-03-30 16:54:00,519 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1645b6b0 to 127.0.0.1:58435 2023-03-30 16:54:00,519 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:00,899 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=756 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/2c445de6e2cd46e1b5586ac20d50f584 2023-03-30 16:54:01,310 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=756 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/b65cd8d9d59745aeaf1c34b309868181 2023-03-30 16:54:01,316 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/51b543c860db4264bf71972ba0283eae as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/51b543c860db4264bf71972ba0283eae 2023-03-30 16:54:01,321 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/51b543c860db4264bf71972ba0283eae, entries=150, sequenceid=756, filesize=11.9 K 2023-03-30 16:54:01,322 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/2c445de6e2cd46e1b5586ac20d50f584 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/2c445de6e2cd46e1b5586ac20d50f584 2023-03-30 16:54:01,326 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/2c445de6e2cd46e1b5586ac20d50f584, entries=150, sequenceid=756, filesize=11.9 K 2023-03-30 16:54:01,327 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/b65cd8d9d59745aeaf1c34b309868181 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/b65cd8d9d59745aeaf1c34b309868181 2023-03-30 16:54:01,332 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/b65cd8d9d59745aeaf1c34b309868181, entries=150, sequenceid=756, filesize=11.9 K 2023-03-30 16:54:01,333 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~425.11 KB/435312, currentSize=20.13 KB/20610 for 4600ee2f6bff5cc4f7b8bf671d05894d in 854ms, sequenceid=756, compaction requested=true 2023-03-30 16:54:01,333 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:54:01,333 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:01,333 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:01,333 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:54:01,333 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:01,333 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:54:01,334 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:54:01,334 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/A is initiating minor compaction (all files) 2023-03-30 16:54:01,334 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:54:01,334 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/A in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:54:01,335 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/B is initiating minor compaction (all files) 2023-03-30 16:54:01,335 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/44382a4bcb684ec0879488c90a2d7ce9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/d18ba9fb18f846e5820b736a11c1796c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/e5a524d57e864563995f40ac28121df1, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/51b543c860db4264bf71972ba0283eae] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=48.4 K 2023-03-30 16:54:01,335 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/B in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:54:01,335 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/4c12935536d848caafb14362b30b9581, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/9692fc75c84440b1a347cca8b519c8f5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/51575202ea8a4663892d436cdd48faa6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/2c445de6e2cd46e1b5586ac20d50f584] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=48.4 K 2023-03-30 16:54:01,335 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 44382a4bcb684ec0879488c90a2d7ce9, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680195236765 2023-03-30 16:54:01,335 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 4c12935536d848caafb14362b30b9581, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680195236765 2023-03-30 16:54:01,335 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting d18ba9fb18f846e5820b736a11c1796c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1680195237402 2023-03-30 16:54:01,336 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 9692fc75c84440b1a347cca8b519c8f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1680195237402 2023-03-30 16:54:01,336 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting e5a524d57e864563995f40ac28121df1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1680195237710 2023-03-30 16:54:01,336 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 51575202ea8a4663892d436cdd48faa6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1680195237710 2023-03-30 16:54:01,336 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 51b543c860db4264bf71972ba0283eae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=756, earliestPutTs=1680195238374 2023-03-30 16:54:01,337 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 2c445de6e2cd46e1b5586ac20d50f584, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=756, earliestPutTs=1680195238374 2023-03-30 16:54:01,347 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#A#compaction#87 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:01,347 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#B#compaction#88 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:01,759 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/9380ea5bccee4895a0a3bca5c8482885 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/9380ea5bccee4895a0a3bca5c8482885 2023-03-30 16:54:01,763 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/5545cd23e19d43cc86729b549c6b5390 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/5545cd23e19d43cc86729b549c6b5390 2023-03-30 16:54:01,765 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/A of 4600ee2f6bff5cc4f7b8bf671d05894d into 9380ea5bccee4895a0a3bca5c8482885(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:01,765 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:54:01,765 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/A, priority=12, startTime=1680195241333; duration=0sec 2023-03-30 16:54:01,765 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:01,765 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:54:01,766 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:54:01,767 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4600ee2f6bff5cc4f7b8bf671d05894d/C is initiating minor compaction (all files) 2023-03-30 16:54:01,767 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4600ee2f6bff5cc4f7b8bf671d05894d/C in TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:54:01,767 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/72ba2ad449a84dc29906282d89e21ca1, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/52495e1f61564c7f8d4d1993612712eb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/4bd0ea2a5ccb4d5287df545cd1ca19a9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/b65cd8d9d59745aeaf1c34b309868181] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp, totalSize=48.4 K 2023-03-30 16:54:01,767 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 72ba2ad449a84dc29906282d89e21ca1, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680195236765 2023-03-30 16:54:01,768 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 52495e1f61564c7f8d4d1993612712eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1680195237402 2023-03-30 16:54:01,768 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 4bd0ea2a5ccb4d5287df545cd1ca19a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1680195237710 2023-03-30 16:54:01,768 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/B of 4600ee2f6bff5cc4f7b8bf671d05894d into 5545cd23e19d43cc86729b549c6b5390(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:01,768 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:54:01,769 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/B, priority=12, startTime=1680195241333; duration=0sec 2023-03-30 16:54:01,769 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:01,769 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting b65cd8d9d59745aeaf1c34b309868181, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=756, earliestPutTs=1680195238374 2023-03-30 16:54:01,778 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4600ee2f6bff5cc4f7b8bf671d05894d#C#compaction#89 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:02,188 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/52bcaa54792c48c19d649afa2b8cbf6c as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/52bcaa54792c48c19d649afa2b8cbf6c 2023-03-30 16:54:02,194 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4600ee2f6bff5cc4f7b8bf671d05894d/C of 4600ee2f6bff5cc4f7b8bf671d05894d into 52bcaa54792c48c19d649afa2b8cbf6c(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:02,194 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:54:02,194 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d., storeName=4600ee2f6bff5cc4f7b8bf671d05894d/C, priority=12, startTime=1680195241333; duration=0sec 2023-03-30 16:54:02,194 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:03,471 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x72ea4a65 to 127.0.0.1:58435 2023-03-30 16:54:03,471 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:03,471 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-30 16:54:03,471 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 164 2023-03-30 16:54:03,471 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 126 2023-03-30 16:54:03,471 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 134 2023-03-30 16:54:03,471 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 119 2023-03-30 16:54:03,471 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 127 2023-03-30 16:54:03,472 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-30 16:54:03,472 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(398): read 4512 2023-03-30 16:54:03,472 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(398): read 4486 2023-03-30 16:54:03,472 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-30 16:54:03,472 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(402): scanned 1997 2023-03-30 16:54:03,472 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(403): verified 5990 rows 2023-03-30 16:54:03,472 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(402): scanned 1987 2023-03-30 16:54:03,472 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(403): verified 5957 rows 2023-03-30 16:54:03,472 INFO [Listener at localhost/38075] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-30 16:54:03,472 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0013510a to 127.0.0.1:58435 2023-03-30 16:54:03,472 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:03,479 INFO [Listener at localhost/38075] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-30 16:54:03,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-30 16:54:03,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-30 16:54:03,493 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195243493"}]},"ts":"1680195243493"} 2023-03-30 16:54:03,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-30 16:54:03,494 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-30 16:54:03,497 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-30 16:54:03,499 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4600ee2f6bff5cc4f7b8bf671d05894d, UNASSIGN}] 2023-03-30 16:54:03,500 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4600ee2f6bff5cc4f7b8bf671d05894d, UNASSIGN 2023-03-30 16:54:03,501 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=4600ee2f6bff5cc4f7b8bf671d05894d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:03,501 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195243501"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680195243501"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680195243501"}]},"ts":"1680195243501"} 2023-03-30 16:54:03,502 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285}] 2023-03-30 16:54:03,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-30 16:54:03,660 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:54:03,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4600ee2f6bff5cc4f7b8bf671d05894d, disabling compactions & flushes 2023-03-30 16:54:03,660 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:54:03,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:54:03,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. after waiting 0 ms 2023-03-30 16:54:03,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:54:03,660 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 4600ee2f6bff5cc4f7b8bf671d05894d 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-30 16:54:03,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=A 2023-03-30 16:54:03,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:03,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=B 2023-03-30 16:54:03,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:03,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4600ee2f6bff5cc4f7b8bf671d05894d, store=C 2023-03-30 16:54:03,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:03,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-30 16:54:04,068 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/1e8d97fe89cc4871b8d3d8cc3c3dc026 2023-03-30 16:54:04,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-30 16:54:04,480 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/e565c921bff546afb8774aeb9feebb90 2023-03-30 16:54:04,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-30 16:54:04,890 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/5aa7c6a82fb948c880a6cb6d975ed75d 2023-03-30 16:54:04,895 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/A/1e8d97fe89cc4871b8d3d8cc3c3dc026 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/1e8d97fe89cc4871b8d3d8cc3c3dc026 2023-03-30 16:54:04,899 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/1e8d97fe89cc4871b8d3d8cc3c3dc026, entries=150, sequenceid=766, filesize=11.9 K 2023-03-30 16:54:04,900 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/B/e565c921bff546afb8774aeb9feebb90 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/e565c921bff546afb8774aeb9feebb90 2023-03-30 16:54:04,905 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/e565c921bff546afb8774aeb9feebb90, entries=150, sequenceid=766, filesize=11.9 K 2023-03-30 16:54:04,906 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/.tmp/C/5aa7c6a82fb948c880a6cb6d975ed75d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/5aa7c6a82fb948c880a6cb6d975ed75d 2023-03-30 16:54:04,910 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/5aa7c6a82fb948c880a6cb6d975ed75d, entries=150, sequenceid=766, filesize=11.9 K 2023-03-30 16:54:04,910 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 4600ee2f6bff5cc4f7b8bf671d05894d in 1250ms, sequenceid=766, compaction requested=false 2023-03-30 16:54:04,930 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/29b27f783da2410890836b3acf842c73, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/f31b411b148a4f7fb23bab8977c8e89a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/2fe13d19e14e49c49f5e1505655faa14, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/91bbbca69cd74145ace260509072a55f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/5d9e9067be924258b1f917729735c8ff, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/b5f1f53a22e84398a2c81d9ba1348681, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/448d6b0211304d54b05776103112d83d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/6b0f00f241a44e2c88779998a8be8271, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/0b23d0496553497cba38d70407bb16b1, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/60a786b3c1fd46f8aec2670522f3e4a7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/00b8445b390e434dbbf2f159a66adcac, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/520cf7a76ff24bc785d8e9890feebad6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/fdddefd957304fefbd84d5745910e2c7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/237b877eac3e47399e04dd4611c30ac2, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/9c61a835c1cf4827b9a17db31ba5534d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/dc6e78c02b1d41abbb512cfb0fcd20eb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/9a991396fbf24827a335f893269eab57, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/89c391536c5c4295bcd4a7aad2bc95e1, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/19758b1e67d647f7b280126ff49d9e4a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/a4de89ceb9c54479aed074961bd90d4d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/e403e416f45846458ad9f55ef0b994ac, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/294d94830cdb49a3afd6151522803695, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/208f483511624e1e8495a1c43d70442f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/f6f6ff705bbb4cb392b29549690d2aa5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/44382a4bcb684ec0879488c90a2d7ce9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/9566ec4c0e7c4de0a69ca143efb4bfaf, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/d18ba9fb18f846e5820b736a11c1796c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/e5a524d57e864563995f40ac28121df1, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/51b543c860db4264bf71972ba0283eae] to archive 2023-03-30 16:54:04,931 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 16:54:04,936 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/29b27f783da2410890836b3acf842c73 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/29b27f783da2410890836b3acf842c73 2023-03-30 16:54:04,938 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/f31b411b148a4f7fb23bab8977c8e89a to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/f31b411b148a4f7fb23bab8977c8e89a 2023-03-30 16:54:04,939 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/2fe13d19e14e49c49f5e1505655faa14 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/2fe13d19e14e49c49f5e1505655faa14 2023-03-30 16:54:04,940 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/91bbbca69cd74145ace260509072a55f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/91bbbca69cd74145ace260509072a55f 2023-03-30 16:54:04,942 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/5d9e9067be924258b1f917729735c8ff to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/5d9e9067be924258b1f917729735c8ff 2023-03-30 16:54:04,943 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/b5f1f53a22e84398a2c81d9ba1348681 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/b5f1f53a22e84398a2c81d9ba1348681 2023-03-30 16:54:04,944 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/448d6b0211304d54b05776103112d83d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/448d6b0211304d54b05776103112d83d 2023-03-30 16:54:04,946 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/6b0f00f241a44e2c88779998a8be8271 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/6b0f00f241a44e2c88779998a8be8271 2023-03-30 16:54:04,947 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/0b23d0496553497cba38d70407bb16b1 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/0b23d0496553497cba38d70407bb16b1 2023-03-30 16:54:04,948 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/60a786b3c1fd46f8aec2670522f3e4a7 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/60a786b3c1fd46f8aec2670522f3e4a7 2023-03-30 16:54:04,949 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/00b8445b390e434dbbf2f159a66adcac to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/00b8445b390e434dbbf2f159a66adcac 2023-03-30 16:54:04,950 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/520cf7a76ff24bc785d8e9890feebad6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/520cf7a76ff24bc785d8e9890feebad6 2023-03-30 16:54:04,951 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/fdddefd957304fefbd84d5745910e2c7 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/fdddefd957304fefbd84d5745910e2c7 2023-03-30 16:54:04,953 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/237b877eac3e47399e04dd4611c30ac2 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/237b877eac3e47399e04dd4611c30ac2 2023-03-30 16:54:04,954 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/9c61a835c1cf4827b9a17db31ba5534d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/9c61a835c1cf4827b9a17db31ba5534d 2023-03-30 16:54:04,956 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/dc6e78c02b1d41abbb512cfb0fcd20eb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/dc6e78c02b1d41abbb512cfb0fcd20eb 2023-03-30 16:54:04,957 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/9a991396fbf24827a335f893269eab57 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/9a991396fbf24827a335f893269eab57 2023-03-30 16:54:04,959 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/89c391536c5c4295bcd4a7aad2bc95e1 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/89c391536c5c4295bcd4a7aad2bc95e1 2023-03-30 16:54:04,960 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/19758b1e67d647f7b280126ff49d9e4a to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/19758b1e67d647f7b280126ff49d9e4a 2023-03-30 16:54:04,961 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/a4de89ceb9c54479aed074961bd90d4d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/a4de89ceb9c54479aed074961bd90d4d 2023-03-30 16:54:04,963 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/e403e416f45846458ad9f55ef0b994ac to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/e403e416f45846458ad9f55ef0b994ac 2023-03-30 16:54:04,964 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/294d94830cdb49a3afd6151522803695 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/294d94830cdb49a3afd6151522803695 2023-03-30 16:54:04,965 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/208f483511624e1e8495a1c43d70442f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/208f483511624e1e8495a1c43d70442f 2023-03-30 16:54:04,967 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/f6f6ff705bbb4cb392b29549690d2aa5 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/f6f6ff705bbb4cb392b29549690d2aa5 2023-03-30 16:54:04,968 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/44382a4bcb684ec0879488c90a2d7ce9 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/44382a4bcb684ec0879488c90a2d7ce9 2023-03-30 16:54:04,969 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/9566ec4c0e7c4de0a69ca143efb4bfaf to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/9566ec4c0e7c4de0a69ca143efb4bfaf 2023-03-30 16:54:04,970 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/d18ba9fb18f846e5820b736a11c1796c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/d18ba9fb18f846e5820b736a11c1796c 2023-03-30 16:54:04,972 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/e5a524d57e864563995f40ac28121df1 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/e5a524d57e864563995f40ac28121df1 2023-03-30 16:54:04,973 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/51b543c860db4264bf71972ba0283eae to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/51b543c860db4264bf71972ba0283eae 2023-03-30 16:54:04,998 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/82c8c3a2310548479c242a44396d5704, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/a6701ff30abc435280e99068521b43b6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/ba389fa14dde4064a070b904b13a7509, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/fcb762bb03404bbc9c16c5646ba66d97, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/acecd5442a3c4e5dba63a830513000a5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/870f5cd957474240bb711a0edf7f55b9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/dfcbb9c1c7b44f299655775c25be503f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/f0eda55a7c2a4759a3dc770a3247d10b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/2f98af72f6084334b72e52b9824359e8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/261cda4e0e2d4e40aed7d1e2c8d5e7f8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/3c5eb3ff345a4c02a1c3aff97bba8a46, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/2e9ea38d207f47588284da2da2a46172, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/86f9209ec63b48c7a1b65bd993ec8823, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/640fa9c6d2d943b59fc840c092cf340b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/084e3dad8cb545738861f2825de6deb0, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/a7b482e81c524d679d4b9e739b276611, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/ba9c6b4839e04ea0a87814c455351f58, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/6832861d49be49e09090bb75cf800d4c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/6fc28b6fa6ff4040a3d3466ceab0859b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/65fb2d3a9455400987c0899ea2bd6bd0, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/e4f4223f3d4e49cdbf36df811c4bb0e2, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/90842eea8b334197ae3bb21a0601c555, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/02ca5c017c784549b72b5d9d5bc4ff35, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/79d11945b3ea47b88f8cb6a8eedcb8f9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/4c12935536d848caafb14362b30b9581, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/61d01981a16d45359ac2897fe552b8b7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/9692fc75c84440b1a347cca8b519c8f5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/51575202ea8a4663892d436cdd48faa6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/2c445de6e2cd46e1b5586ac20d50f584] to archive 2023-03-30 16:54:04,999 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 16:54:05,001 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/82c8c3a2310548479c242a44396d5704 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/82c8c3a2310548479c242a44396d5704 2023-03-30 16:54:05,002 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/a6701ff30abc435280e99068521b43b6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/a6701ff30abc435280e99068521b43b6 2023-03-30 16:54:05,003 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/ba389fa14dde4064a070b904b13a7509 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/ba389fa14dde4064a070b904b13a7509 2023-03-30 16:54:05,004 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/fcb762bb03404bbc9c16c5646ba66d97 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/fcb762bb03404bbc9c16c5646ba66d97 2023-03-30 16:54:05,005 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/acecd5442a3c4e5dba63a830513000a5 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/acecd5442a3c4e5dba63a830513000a5 2023-03-30 16:54:05,007 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/870f5cd957474240bb711a0edf7f55b9 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/870f5cd957474240bb711a0edf7f55b9 2023-03-30 16:54:05,008 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/dfcbb9c1c7b44f299655775c25be503f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/dfcbb9c1c7b44f299655775c25be503f 2023-03-30 16:54:05,009 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/f0eda55a7c2a4759a3dc770a3247d10b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/f0eda55a7c2a4759a3dc770a3247d10b 2023-03-30 16:54:05,010 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/2f98af72f6084334b72e52b9824359e8 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/2f98af72f6084334b72e52b9824359e8 2023-03-30 16:54:05,011 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/261cda4e0e2d4e40aed7d1e2c8d5e7f8 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/261cda4e0e2d4e40aed7d1e2c8d5e7f8 2023-03-30 16:54:05,012 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/3c5eb3ff345a4c02a1c3aff97bba8a46 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/3c5eb3ff345a4c02a1c3aff97bba8a46 2023-03-30 16:54:05,014 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/2e9ea38d207f47588284da2da2a46172 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/2e9ea38d207f47588284da2da2a46172 2023-03-30 16:54:05,015 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/86f9209ec63b48c7a1b65bd993ec8823 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/86f9209ec63b48c7a1b65bd993ec8823 2023-03-30 16:54:05,016 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/640fa9c6d2d943b59fc840c092cf340b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/640fa9c6d2d943b59fc840c092cf340b 2023-03-30 16:54:05,017 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/084e3dad8cb545738861f2825de6deb0 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/084e3dad8cb545738861f2825de6deb0 2023-03-30 16:54:05,019 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/a7b482e81c524d679d4b9e739b276611 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/a7b482e81c524d679d4b9e739b276611 2023-03-30 16:54:05,020 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/ba9c6b4839e04ea0a87814c455351f58 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/ba9c6b4839e04ea0a87814c455351f58 2023-03-30 16:54:05,021 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/6832861d49be49e09090bb75cf800d4c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/6832861d49be49e09090bb75cf800d4c 2023-03-30 16:54:05,022 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/6fc28b6fa6ff4040a3d3466ceab0859b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/6fc28b6fa6ff4040a3d3466ceab0859b 2023-03-30 16:54:05,023 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/65fb2d3a9455400987c0899ea2bd6bd0 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/65fb2d3a9455400987c0899ea2bd6bd0 2023-03-30 16:54:05,025 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/e4f4223f3d4e49cdbf36df811c4bb0e2 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/e4f4223f3d4e49cdbf36df811c4bb0e2 2023-03-30 16:54:05,026 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/90842eea8b334197ae3bb21a0601c555 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/90842eea8b334197ae3bb21a0601c555 2023-03-30 16:54:05,027 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/02ca5c017c784549b72b5d9d5bc4ff35 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/02ca5c017c784549b72b5d9d5bc4ff35 2023-03-30 16:54:05,028 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/79d11945b3ea47b88f8cb6a8eedcb8f9 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/79d11945b3ea47b88f8cb6a8eedcb8f9 2023-03-30 16:54:05,029 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/4c12935536d848caafb14362b30b9581 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/4c12935536d848caafb14362b30b9581 2023-03-30 16:54:05,031 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/61d01981a16d45359ac2897fe552b8b7 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/61d01981a16d45359ac2897fe552b8b7 2023-03-30 16:54:05,032 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/9692fc75c84440b1a347cca8b519c8f5 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/9692fc75c84440b1a347cca8b519c8f5 2023-03-30 16:54:05,033 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/51575202ea8a4663892d436cdd48faa6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/51575202ea8a4663892d436cdd48faa6 2023-03-30 16:54:05,034 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/2c445de6e2cd46e1b5586ac20d50f584 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/2c445de6e2cd46e1b5586ac20d50f584 2023-03-30 16:54:05,037 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/a31fbccbf633462ab419b9448c8380df, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/4d1ec7888d0c40809b256b0bb0ad6e8a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/b484898c087d4f14bc44df334024faf5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/98a966eb1c0a4488a9ebdb34df590fff, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/26d909c968cb41958c7a053c0e442382, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/6e0d61e1736446878914529cdbbe9d05, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/10d26e892f814870a3f79786f9200a13, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/09730aa06f3844c1b4f2a22bcacbf2e4, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/00cc491da8724739aec0da7b6d84085c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/e54d4fdd9ede4b7caa0c5ddad6fa258b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/5da908072b584a7d8693f416ec29fca8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/3f3d18ef30184aedbe3c557358e76f2f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/03f2c4a20209410383c65822e5c5b48b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/810afd450dad4541875bdd16da98f66a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/635d157ae0394d94911bf0a13135ce84, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/affd0070495e49a19172de13a0652541, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/a0809f3e316e401e99c3de62e3d6cd69, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/cd1373f64fb94637ac2f8fd44472d88b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/e65aa8f539f549e5b0a3587ae9f2f706, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/1d377189fb434958aae987491313e42c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/fc7f0739b19043839d979125844b578d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/0880d9952c3f4cd69c571c3bc0d419f2, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/afa6aa966b9a45b0ad74abfe25d94045, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/11f465ff1a724206bbc02b07d92876bb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/72ba2ad449a84dc29906282d89e21ca1, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/4a6baecd9d004cbcb0a52da252c2c2ac, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/52495e1f61564c7f8d4d1993612712eb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/4bd0ea2a5ccb4d5287df545cd1ca19a9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/b65cd8d9d59745aeaf1c34b309868181] to archive 2023-03-30 16:54:05,038 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 16:54:05,039 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/a31fbccbf633462ab419b9448c8380df to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/a31fbccbf633462ab419b9448c8380df 2023-03-30 16:54:05,041 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/4d1ec7888d0c40809b256b0bb0ad6e8a to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/4d1ec7888d0c40809b256b0bb0ad6e8a 2023-03-30 16:54:05,042 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/b484898c087d4f14bc44df334024faf5 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/b484898c087d4f14bc44df334024faf5 2023-03-30 16:54:05,043 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/98a966eb1c0a4488a9ebdb34df590fff to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/98a966eb1c0a4488a9ebdb34df590fff 2023-03-30 16:54:05,044 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/26d909c968cb41958c7a053c0e442382 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/26d909c968cb41958c7a053c0e442382 2023-03-30 16:54:05,045 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/6e0d61e1736446878914529cdbbe9d05 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/6e0d61e1736446878914529cdbbe9d05 2023-03-30 16:54:05,046 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/10d26e892f814870a3f79786f9200a13 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/10d26e892f814870a3f79786f9200a13 2023-03-30 16:54:05,047 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/09730aa06f3844c1b4f2a22bcacbf2e4 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/09730aa06f3844c1b4f2a22bcacbf2e4 2023-03-30 16:54:05,048 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/00cc491da8724739aec0da7b6d84085c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/00cc491da8724739aec0da7b6d84085c 2023-03-30 16:54:05,049 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/e54d4fdd9ede4b7caa0c5ddad6fa258b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/e54d4fdd9ede4b7caa0c5ddad6fa258b 2023-03-30 16:54:05,050 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/5da908072b584a7d8693f416ec29fca8 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/5da908072b584a7d8693f416ec29fca8 2023-03-30 16:54:05,051 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/3f3d18ef30184aedbe3c557358e76f2f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/3f3d18ef30184aedbe3c557358e76f2f 2023-03-30 16:54:05,052 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/03f2c4a20209410383c65822e5c5b48b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/03f2c4a20209410383c65822e5c5b48b 2023-03-30 16:54:05,053 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/810afd450dad4541875bdd16da98f66a to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/810afd450dad4541875bdd16da98f66a 2023-03-30 16:54:05,054 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/635d157ae0394d94911bf0a13135ce84 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/635d157ae0394d94911bf0a13135ce84 2023-03-30 16:54:05,055 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/affd0070495e49a19172de13a0652541 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/affd0070495e49a19172de13a0652541 2023-03-30 16:54:05,057 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/a0809f3e316e401e99c3de62e3d6cd69 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/a0809f3e316e401e99c3de62e3d6cd69 2023-03-30 16:54:05,058 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/cd1373f64fb94637ac2f8fd44472d88b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/cd1373f64fb94637ac2f8fd44472d88b 2023-03-30 16:54:05,059 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/e65aa8f539f549e5b0a3587ae9f2f706 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/e65aa8f539f549e5b0a3587ae9f2f706 2023-03-30 16:54:05,060 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/1d377189fb434958aae987491313e42c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/1d377189fb434958aae987491313e42c 2023-03-30 16:54:05,061 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/fc7f0739b19043839d979125844b578d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/fc7f0739b19043839d979125844b578d 2023-03-30 16:54:05,062 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/0880d9952c3f4cd69c571c3bc0d419f2 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/0880d9952c3f4cd69c571c3bc0d419f2 2023-03-30 16:54:05,063 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/afa6aa966b9a45b0ad74abfe25d94045 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/afa6aa966b9a45b0ad74abfe25d94045 2023-03-30 16:54:05,064 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/11f465ff1a724206bbc02b07d92876bb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/11f465ff1a724206bbc02b07d92876bb 2023-03-30 16:54:05,065 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/72ba2ad449a84dc29906282d89e21ca1 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/72ba2ad449a84dc29906282d89e21ca1 2023-03-30 16:54:05,066 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/4a6baecd9d004cbcb0a52da252c2c2ac to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/4a6baecd9d004cbcb0a52da252c2c2ac 2023-03-30 16:54:05,067 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/52495e1f61564c7f8d4d1993612712eb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/52495e1f61564c7f8d4d1993612712eb 2023-03-30 16:54:05,068 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/4bd0ea2a5ccb4d5287df545cd1ca19a9 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/4bd0ea2a5ccb4d5287df545cd1ca19a9 2023-03-30 16:54:05,069 DEBUG [StoreCloser-TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/b65cd8d9d59745aeaf1c34b309868181 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/b65cd8d9d59745aeaf1c34b309868181 2023-03-30 16:54:05,074 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/recovered.edits/769.seqid, newMaxSeqId=769, maxSeqId=1 2023-03-30 16:54:05,076 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d. 2023-03-30 16:54:05,076 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4600ee2f6bff5cc4f7b8bf671d05894d: 2023-03-30 16:54:05,078 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:54:05,078 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=4600ee2f6bff5cc4f7b8bf671d05894d, regionState=CLOSED 2023-03-30 16:54:05,079 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195245078"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680195245078"}]},"ts":"1680195245078"} 2023-03-30 16:54:05,081 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-03-30 16:54:05,082 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 4600ee2f6bff5cc4f7b8bf671d05894d, server=jenkins-hbase4.apache.org,45193,1680195214285 in 1.5780 sec 2023-03-30 16:54:05,083 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-03-30 16:54:05,083 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4600ee2f6bff5cc4f7b8bf671d05894d, UNASSIGN in 1.5820 sec 2023-03-30 16:54:05,084 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195245084"}]},"ts":"1680195245084"} 2023-03-30 16:54:05,085 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-30 16:54:05,088 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-30 16:54:05,090 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6030 sec 2023-03-30 16:54:05,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-30 16:54:05,599 INFO [Listener at localhost/38075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-03-30 16:54:05,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-30 16:54:05,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:54:05,610 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:54:05,612 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:54:05,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-30 16:54:05,616 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:54:05,619 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A, FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B, FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C, FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/recovered.edits] 2023-03-30 16:54:05,622 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/1e8d97fe89cc4871b8d3d8cc3c3dc026 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/1e8d97fe89cc4871b8d3d8cc3c3dc026 2023-03-30 16:54:05,623 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/9380ea5bccee4895a0a3bca5c8482885 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/A/9380ea5bccee4895a0a3bca5c8482885 2023-03-30 16:54:05,625 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/5545cd23e19d43cc86729b549c6b5390 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/5545cd23e19d43cc86729b549c6b5390 2023-03-30 16:54:05,627 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/e565c921bff546afb8774aeb9feebb90 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/B/e565c921bff546afb8774aeb9feebb90 2023-03-30 16:54:05,629 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/52bcaa54792c48c19d649afa2b8cbf6c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/52bcaa54792c48c19d649afa2b8cbf6c 2023-03-30 16:54:05,631 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/5aa7c6a82fb948c880a6cb6d975ed75d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/C/5aa7c6a82fb948c880a6cb6d975ed75d 2023-03-30 16:54:05,634 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/recovered.edits/769.seqid to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d/recovered.edits/769.seqid 2023-03-30 16:54:05,634 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4600ee2f6bff5cc4f7b8bf671d05894d 2023-03-30 16:54:05,634 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 16:54:05,637 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:54:05,659 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-30 16:54:05,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-30 16:54:05,718 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-30 16:54:05,720 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:54:05,720 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-30 16:54:05,720 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680195245720"}]},"ts":"9223372036854775807"} 2023-03-30 16:54:05,722 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-30 16:54:05,722 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 4600ee2f6bff5cc4f7b8bf671d05894d, NAME => 'TestAcidGuarantees,,1680195218023.4600ee2f6bff5cc4f7b8bf671d05894d.', STARTKEY => '', ENDKEY => ''}] 2023-03-30 16:54:05,722 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-30 16:54:05,723 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680195245722"}]},"ts":"9223372036854775807"} 2023-03-30 16:54:05,724 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-30 16:54:05,727 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:54:05,728 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 123 msec 2023-03-30 16:54:05,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-30 16:54:05,916 INFO [Listener at localhost/38075] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-03-30 16:54:05,932 INFO [Listener at localhost/38075] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=234 (was 206) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:45193-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_288265338_15 at /127.0.0.1:44656 [Waiting for operation #7] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4f91ae25-shared-pool-27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4f91ae25-shared-pool-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4f91ae25-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4f91ae25-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_288265338_15 at /127.0.0.1:46692 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=579 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=349 (was 307) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=4828 (was 5471) 2023-03-30 16:54:05,946 INFO [Listener at localhost/38075] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=234, OpenFileDescriptor=579, MaxFileDescriptor=60000, SystemLoadAverage=349, ProcessCount=169, AvailableMemoryMB=4827 2023-03-30 16:54:05,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-30 16:54:05,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-30 16:54:05,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-30 16:54:05,951 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-30 16:54:05,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-03-30 16:54:05,952 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-30 16:54:05,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-30 16:54:05,954 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:05,954 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47 empty. 2023-03-30 16:54:05,955 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:05,955 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 16:54:06,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-30 16:54:06,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-30 16:54:06,365 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-30 16:54:06,366 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 339a657579649f1a81384cc0c3fc4f47, NAME => 'TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp 2023-03-30 16:54:06,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-30 16:54:06,773 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:54:06,773 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 339a657579649f1a81384cc0c3fc4f47, disabling compactions & flushes 2023-03-30 16:54:06,773 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:06,773 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:06,773 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. after waiting 0 ms 2023-03-30 16:54:06,773 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:06,773 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:06,773 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:06,776 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-30 16:54:06,777 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195246777"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680195246777"}]},"ts":"1680195246777"} 2023-03-30 16:54:06,779 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-30 16:54:06,780 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-30 16:54:06,780 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195246780"}]},"ts":"1680195246780"} 2023-03-30 16:54:06,781 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-30 16:54:06,785 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=339a657579649f1a81384cc0c3fc4f47, ASSIGN}] 2023-03-30 16:54:06,786 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=339a657579649f1a81384cc0c3fc4f47, ASSIGN 2023-03-30 16:54:06,787 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=339a657579649f1a81384cc0c3fc4f47, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,45193,1680195214285; forceNewPlan=false, retain=false 2023-03-30 16:54:06,938 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=339a657579649f1a81384cc0c3fc4f47, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:06,938 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195246938"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680195246938"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680195246938"}]},"ts":"1680195246938"} 2023-03-30 16:54:06,940 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285}] 2023-03-30 16:54:07,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-30 16:54:07,096 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:07,096 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 339a657579649f1a81384cc0c3fc4f47, NAME => 'TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.', STARTKEY => '', ENDKEY => ''} 2023-03-30 16:54:07,097 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:07,097 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:54:07,097 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:07,097 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:07,098 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:07,100 DEBUG [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A 2023-03-30 16:54:07,100 DEBUG [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A 2023-03-30 16:54:07,100 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:54:07,100 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 339a657579649f1a81384cc0c3fc4f47 columnFamilyName A 2023-03-30 16:54:07,101 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] regionserver.HStore(310): Store=339a657579649f1a81384cc0c3fc4f47/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:54:07,101 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:07,102 DEBUG [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B 2023-03-30 16:54:07,103 DEBUG [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B 2023-03-30 16:54:07,103 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:54:07,103 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 339a657579649f1a81384cc0c3fc4f47 columnFamilyName B 2023-03-30 16:54:07,104 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] regionserver.HStore(310): Store=339a657579649f1a81384cc0c3fc4f47/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:54:07,104 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:07,105 DEBUG [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C 2023-03-30 16:54:07,105 DEBUG [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C 2023-03-30 16:54:07,105 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:54:07,106 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 339a657579649f1a81384cc0c3fc4f47 columnFamilyName C 2023-03-30 16:54:07,106 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] regionserver.HStore(310): Store=339a657579649f1a81384cc0c3fc4f47/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:54:07,106 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:07,107 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:07,107 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:07,109 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 16:54:07,110 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:07,112 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-30 16:54:07,112 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 339a657579649f1a81384cc0c3fc4f47; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62232759, jitterRate=-0.07265962660312653}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 16:54:07,113 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:07,113 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., pid=18, masterSystemTime=1680195247092 2023-03-30 16:54:07,115 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:07,115 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:07,115 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=339a657579649f1a81384cc0c3fc4f47, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:07,115 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195247115"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680195247115"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680195247115"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680195247115"}]},"ts":"1680195247115"} 2023-03-30 16:54:07,118 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-03-30 16:54:07,118 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 in 177 msec 2023-03-30 16:54:07,120 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-03-30 16:54:07,120 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=339a657579649f1a81384cc0c3fc4f47, ASSIGN in 333 msec 2023-03-30 16:54:07,121 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-30 16:54:07,121 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195247121"}]},"ts":"1680195247121"} 2023-03-30 16:54:07,122 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-30 16:54:07,124 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-30 16:54:07,126 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1770 sec 2023-03-30 16:54:08,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-30 16:54:08,057 INFO [Listener at localhost/38075] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-03-30 16:54:08,058 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x58d22e5a to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:08,065 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4910cda, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:08,066 DEBUG [Listener at localhost/38075] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:08,071 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46754, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:08,072 DEBUG [Listener at localhost/38075] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-30 16:54:08,075 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56020, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-30 16:54:08,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-30 16:54:08,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-30 16:54:08,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-30 16:54:08,504 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-30 16:54:08,505 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-30 16:54:08,505 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-30 16:54:08,509 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-30 16:54:08,521 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=339a657579649f1a81384cc0c3fc4f47, REOPEN/MOVE}] 2023-03-30 16:54:08,522 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=339a657579649f1a81384cc0c3fc4f47, REOPEN/MOVE 2023-03-30 16:54:08,522 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=339a657579649f1a81384cc0c3fc4f47, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:08,522 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195248522"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680195248522"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680195248522"}]},"ts":"1680195248522"} 2023-03-30 16:54:08,523 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285}] 2023-03-30 16:54:08,675 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:08,676 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 339a657579649f1a81384cc0c3fc4f47, disabling compactions & flushes 2023-03-30 16:54:08,676 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:08,676 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:08,676 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. after waiting 0 ms 2023-03-30 16:54:08,676 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:08,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-30 16:54:08,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:08,681 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:08,681 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 339a657579649f1a81384cc0c3fc4f47 to self. 2023-03-30 16:54:08,682 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:08,683 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=339a657579649f1a81384cc0c3fc4f47, regionState=CLOSED 2023-03-30 16:54:08,683 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195248683"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680195248683"}]},"ts":"1680195248683"} 2023-03-30 16:54:08,686 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-03-30 16:54:08,686 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 in 161 msec 2023-03-30 16:54:08,687 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=339a657579649f1a81384cc0c3fc4f47, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,45193,1680195214285; forceNewPlan=false, retain=true 2023-03-30 16:54:08,837 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=339a657579649f1a81384cc0c3fc4f47, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:08,838 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195248837"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680195248837"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680195248837"}]},"ts":"1680195248837"} 2023-03-30 16:54:08,840 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285}] 2023-03-30 16:54:08,995 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:08,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 339a657579649f1a81384cc0c3fc4f47, NAME => 'TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.', STARTKEY => '', ENDKEY => ''} 2023-03-30 16:54:08,996 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:08,996 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:54:08,996 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:08,996 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:08,998 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:08,999 DEBUG [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A 2023-03-30 16:54:08,999 DEBUG [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A 2023-03-30 16:54:09,000 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:54:09,002 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 339a657579649f1a81384cc0c3fc4f47 columnFamilyName A 2023-03-30 16:54:09,004 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] regionserver.HStore(310): Store=339a657579649f1a81384cc0c3fc4f47/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:54:09,005 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:09,006 DEBUG [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B 2023-03-30 16:54:09,006 DEBUG [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B 2023-03-30 16:54:09,006 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:54:09,007 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 339a657579649f1a81384cc0c3fc4f47 columnFamilyName B 2023-03-30 16:54:09,008 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] regionserver.HStore(310): Store=339a657579649f1a81384cc0c3fc4f47/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:54:09,008 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:09,008 DEBUG [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C 2023-03-30 16:54:09,009 DEBUG [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C 2023-03-30 16:54:09,009 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:54:09,010 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 339a657579649f1a81384cc0c3fc4f47 columnFamilyName C 2023-03-30 16:54:09,010 INFO [StoreOpener-339a657579649f1a81384cc0c3fc4f47-1] regionserver.HStore(310): Store=339a657579649f1a81384cc0c3fc4f47/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:54:09,010 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:09,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:09,012 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:09,014 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 16:54:09,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:09,016 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 339a657579649f1a81384cc0c3fc4f47; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73416777, jitterRate=0.09399522840976715}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 16:54:09,016 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:09,016 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., pid=23, masterSystemTime=1680195248991 2023-03-30 16:54:09,018 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:09,018 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:09,018 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=339a657579649f1a81384cc0c3fc4f47, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,018 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195249018"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680195249018"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680195249018"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680195249018"}]},"ts":"1680195249018"} 2023-03-30 16:54:09,021 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-03-30 16:54:09,021 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 in 180 msec 2023-03-30 16:54:09,022 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-03-30 16:54:09,023 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=339a657579649f1a81384cc0c3fc4f47, REOPEN/MOVE in 500 msec 2023-03-30 16:54:09,025 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-03-30 16:54:09,025 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 516 msec 2023-03-30 16:54:09,027 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 941 msec 2023-03-30 16:54:09,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-03-30 16:54:09,036 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x05609e0c to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:09,040 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@542e33ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:09,041 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x31cba94a to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:09,046 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@695af552, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:09,046 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x7adf5e8e to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:09,049 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@16323966, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:09,050 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x52f29016 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:09,054 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@550c7a27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:09,054 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x06f13c14 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:09,058 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40c57efc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:09,058 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x2a5a81b5 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:09,063 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4183389c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:09,064 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x45e59be5 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:09,067 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3bccfdc5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:09,067 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x2668d41d to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:09,071 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fbb4986, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:09,071 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x6b416c44 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:09,075 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b69850d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:09,079 DEBUG [hconnection-0x1bd50d22-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:09,079 DEBUG [hconnection-0x5c36d936-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:09,084 DEBUG [hconnection-0x4d92792d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:09,086 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46760, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:09,086 DEBUG [hconnection-0x45e9a492-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:09,086 DEBUG [hconnection-0x3bce5569-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:09,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:09,086 DEBUG [hconnection-0x2c93e0e9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:09,088 DEBUG [hconnection-0x58d20ac4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:09,088 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46776, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:09,088 DEBUG [hconnection-0x5c15429d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:09,088 DEBUG [hconnection-0x47f1524b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:09,093 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46786, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:09,094 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46826, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:09,102 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46798, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:09,102 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46778, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:09,103 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46836, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:09,103 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46810, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:09,103 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46838, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:09,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:09,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:09,112 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:09,112 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:09,112 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:09,112 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:09,114 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,115 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,118 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,118 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,118 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:09,122 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:09,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:09,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:09,124 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:09,124 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:09,124 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:09,132 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:09,133 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:09,133 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 16:54:09,133 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:09,133 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:09,133 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:09,133 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:09,133 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:09,133 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:09,133 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:09,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:09,133 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:09,140 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:09,140 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:09,141 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:09,141 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:09,141 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,153 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:09,153 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:09,155 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,155 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:09,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:09,156 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:09,156 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:09,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:09,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:09,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:09,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7db1217d[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:09,159 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:09,159 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,160 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,160 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:09,160 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:09,160 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:09,160 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,160 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:09,162 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,162 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:09,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:09,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:09,162 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:09,162 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:09,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:09,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:09,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:09,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:09,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:09,166 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:09,166 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:09,166 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:09,166 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:09,166 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:09,170 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,170 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:09,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:09,170 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:09,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:09,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:09,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:09,171 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:09,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,173 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:09,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,177 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:09,177 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:09,178 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,178 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,178 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:09,178 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:09,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:09,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:09,178 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:09,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:09,178 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:09,178 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:09,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:09,178 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,178 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:09,179 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195249154, max timestamp=1680195249178Num uniques 859; 2023-03-30 16:54:09,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195249154, max timestamp=1680195249178Num uniques 859; 2023-03-30 16:54:09,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:54:09,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:54:09,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195249154, max timestamp=1680195249178Num uniques 859; 2023-03-30 16:54:09,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:09,213 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033049e374ee56dc46f798bb6d8029ab6ad5 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033049e374ee56dc46f798bb6d8029ab6ad5 2023-03-30 16:54:09,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:09,243 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:09,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195309195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195309197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:09,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195309244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:09,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195309247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:09,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195309248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,279 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:09,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:09,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:09,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:09,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:09,301 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:09,301 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:09,301 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:09,301 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:09,302 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,302 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,307 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:09,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:09,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:09,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:09,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:09,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:09,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:09,310 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:09,310 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:09,310 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:09,316 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,319 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,319 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:09,332 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:09,332 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:09,332 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:09,332 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:09,332 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:09,332 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:09,332 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:09,332 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:09,332 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,336 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:09,336 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:09,339 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,339 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:09,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:09,340 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,340 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:09,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:09,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:09,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:09,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:09,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bbae6f3[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:09,342 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:09,342 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,343 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,344 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:09,344 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:09,344 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:09,345 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,345 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:09,346 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:09,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:09,346 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:09,346 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:09,346 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:09,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:09,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:09,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:09,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:09,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:09,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:09,352 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:09,352 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:09,352 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:09,352 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:09,352 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:09,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195309345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:09,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195309349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,354 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,354 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:09,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,354 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:09,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:09,355 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:09,356 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:09,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195309349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:09,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195309350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:09,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195309351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,359 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:09,359 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:09,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:09,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:09,359 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,359 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,359 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,359 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,360 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,360 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,360 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:09,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:09,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:09,360 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:09,360 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:09,360 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:09,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:09,464 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:09,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:09,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:09,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:09,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:09,490 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:09,490 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:09,490 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:09,490 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:09,491 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,491 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,492 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,493 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,493 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:09,496 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:09,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:09,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:09,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:09,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:09,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:09,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:09,498 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:09,498 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:09,498 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:09,499 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:09,499 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:09,500 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:09,500 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:09,501 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:09,501 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:09,501 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:09,501 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:09,501 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,504 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:09,504 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:09,506 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,506 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:09,507 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:09,507 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:09,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:09,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:09,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:09,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78fcc0b9[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:09,511 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:09,511 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,513 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,513 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:09,513 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:09,513 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,513 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:09,513 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:09,516 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:09,516 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:09,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:09,516 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:09,516 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:09,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:09,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:09,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:09,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:09,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:09,519 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:09,519 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:09,519 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:09,519 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:09,519 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:09,521 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,521 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:09,521 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,521 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:09,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:09,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,530 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:09,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,536 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:09,536 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:09,536 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:09,536 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:09,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:09,536 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,536 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,536 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,536 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,536 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,536 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:09,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:09,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:09,537 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:09,537 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:09,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:09,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:09,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195309554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:09,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195309556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:09,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195309559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:09,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195309560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:09,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195309562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,625 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/ff30f19580ae4a7db821c88ecd4dbf4f 2023-03-30 16:54:09,637 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:09,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:09,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:09,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:09,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:09,656 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:09,656 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:09,656 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:09,656 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:09,657 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,657 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,658 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,659 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,659 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:09,660 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:09,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:09,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:09,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:09,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:09,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:09,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:09,672 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:09,672 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:09,672 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:09,674 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:09,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:09,675 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:09,675 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:09,675 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:09,675 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:09,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:09,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:09,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,675 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/71f8120f7bc148c79ec130c304a1d4f5 2023-03-30 16:54:09,692 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:09,692 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:09,696 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,696 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:09,697 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,697 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:09,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:09,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7353010a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:09,699 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:09,699 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,700 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,701 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:09,701 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:09,701 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:09,703 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,703 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:09,704 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,704 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:09,704 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:09,704 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:09,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:09,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:09,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:09,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:09,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:09,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:09,718 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:09,718 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:09,718 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:09,718 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:09,718 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:09,720 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,720 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:09,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,720 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:09,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:09,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:09,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:09,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,724 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:09,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:09,727 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,730 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:09,730 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:09,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:09,731 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,731 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,731 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,731 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,731 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,731 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,731 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:09,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:09,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:09,731 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:09,732 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:09,732 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:09,832 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:09,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:09,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:09,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:09,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:09,848 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:09,848 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:09,848 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:09,848 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:09,849 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,849 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,850 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:09,850 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:09,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:09,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,851 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,851 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:09,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:09,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:09,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:09,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:09,856 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:09,856 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:09,856 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:09,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:09,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195309857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:09,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195309858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:09,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195309863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:09,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195309864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:09,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195309867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,872 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:09,872 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:09,874 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:09,875 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:09,875 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:09,875 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:09,875 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:09,875 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:09,875 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,881 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:09,881 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:09,885 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,885 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:09,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:09,885 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,885 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:09,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:09,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:09,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:09,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:09,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79d7cd4[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:09,891 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:09,891 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,893 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,894 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:09,894 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,894 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:09,894 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:09,894 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:09,896 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:09,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:09,896 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:09,896 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:09,896 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:09,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:09,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:09,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:09,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:09,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:09,901 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:09,901 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:09,901 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:09,901 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:09,901 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:09,904 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,904 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:09,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,904 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:09,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:09,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:09,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:09,905 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:09,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,907 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:09,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:09,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,910 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:09,910 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:09,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:09,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:09,910 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,910 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,911 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:09,911 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:09,911 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,911 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:09,911 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:09,911 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:09,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:09,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:09,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:09,911 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:09,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:09,911 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:09,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:10,011 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:10,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:10,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:10,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:10,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:10,020 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:10,020 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:10,020 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:10,020 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:10,021 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:10,021 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:10,023 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:10,023 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:10,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:10,024 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,024 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:10,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:10,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:10,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:10,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:10,026 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:10,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:10,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:10,035 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:10,035 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:10,035 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:10,035 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:10,035 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:10,035 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:10,035 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:10,036 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:10,036 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:10,036 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:10,036 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:10,036 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,036 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:10,036 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:10,037 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:10,037 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:10,038 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:10,038 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:10,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:10,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:10,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:10,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bdc8faa[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:10,040 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:10,040 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:10,042 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,042 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:10,042 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:10,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:10,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:10,043 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:10,043 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:10,043 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:10,044 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:10,044 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,044 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:10,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:10,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:10,044 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:10,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:10,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:10,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:10,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:10,047 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:10,047 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:10,047 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:10,047 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:10,047 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:10,051 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:10,051 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:10,051 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:10,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:10,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:10,052 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:10,053 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:10,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:10,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,055 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,055 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:10,055 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:10,055 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:10,055 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:10,056 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:10,056 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:10,056 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:10,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:10,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:10,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:10,056 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:10,056 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:10,056 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:10,056 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:10,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:10,156 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:10,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:10,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:10,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:10,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:10,164 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:10,164 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:10,164 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:10,164 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:10,164 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:10,164 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:10,165 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:10,166 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:10,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:10,166 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,166 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:10,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:10,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:10,167 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/6f346196a2a440daaaf88f8981701228 2023-03-30 16:54:10,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:10,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:10,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:10,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:10,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:10,168 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:10,168 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:10,168 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:10,169 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:10,169 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:10,169 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:10,169 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:10,170 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:10,170 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:10,170 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:10,170 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:10,170 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,172 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:10,172 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:10,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/ff30f19580ae4a7db821c88ecd4dbf4f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/ff30f19580ae4a7db821c88ecd4dbf4f 2023-03-30 16:54:10,174 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:10,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:10,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:10,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:10,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:10,176 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,179 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:10,179 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:10,179 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/ff30f19580ae4a7db821c88ecd4dbf4f, entries=150, sequenceid=15, filesize=25.1 K 2023-03-30 16:54:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:10,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:10,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55bb6760[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:10,180 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:10,180 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:10,180 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/71f8120f7bc148c79ec130c304a1d4f5 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/71f8120f7bc148c79ec130c304a1d4f5 2023-03-30 16:54:10,182 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:10,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:10,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:10,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:10,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:10,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:10,183 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,183 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:10,184 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,184 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:10,184 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:10,184 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:10,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/71f8120f7bc148c79ec130c304a1d4f5, entries=150, sequenceid=15, filesize=11.6 K 2023-03-30 16:54:10,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:10,187 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/6f346196a2a440daaaf88f8981701228 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/6f346196a2a440daaaf88f8981701228 2023-03-30 16:54:10,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:10,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:10,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:10,190 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:10,190 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:10,190 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:10,190 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:10,190 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:10,192 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:10,192 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:10,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:10,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:10,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:10,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:10,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:10,192 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:10,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:10,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:10,193 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:10,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:10,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:10,195 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/6f346196a2a440daaaf88f8981701228, entries=150, sequenceid=15, filesize=11.6 K 2023-03-30 16:54:10,195 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:10,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:10,196 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=131.94 KB/135110 for 339a657579649f1a81384cc0c3fc4f47 in 1063ms, sequenceid=15, compaction requested=false 2023-03-30 16:54:10,196 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-30 16:54:10,197 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:10,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,202 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,203 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:10,203 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:10,203 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:10,203 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:10,203 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,203 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:10,203 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:10,203 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:10,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:10,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:10,203 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:10,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:10,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:10,203 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:10,203 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:10,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:10,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:10,303 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:10,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:10,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:10,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:10,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:10,308 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:10,308 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:10,308 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:10,308 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:10,308 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:10,308 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:10,310 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,310 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:10,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:10,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:10,310 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,310 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:10,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:10,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:10,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:10,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:10,312 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:10,312 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:10,312 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:10,312 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:10,312 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:10,312 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:10,312 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:10,313 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=131.94 KB heapSize=375.22 KB 2023-03-30 16:54:10,313 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:10,313 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:10,313 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:10,313 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:10,313 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:10,313 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:10,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:10,367 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:10,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:10,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195310369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:10,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195310370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:10,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195310371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:10,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195310372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:10,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195310372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:10,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195310474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:10,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195310477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:10,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195310477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:10,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195310477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,618 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-30 16:54:10,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:10,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195310678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:10,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195310680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:10,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:10,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195310681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195310681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,733 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330c3038b15d77a4d92aa4a2a338456cdb9 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330c3038b15d77a4d92aa4a2a338456cdb9 2023-03-30 16:54:10,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:10,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195310982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:10,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195310984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:10,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195310987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:10,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:10,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195310988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:11,142 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/76934ce5c2b041f3942bf3fdf0f2f1a8 2023-03-30 16:54:11,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:11,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195311381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:11,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:11,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195311487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:11,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:11,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195311488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:11,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:11,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195311488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:11,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:11,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195311495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:11,556 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/fb0e085b67474bc48927096f115f44cc 2023-03-30 16:54:11,974 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/c6b6669a87be44ddac08518de61bb8f4 2023-03-30 16:54:11,979 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/76934ce5c2b041f3942bf3fdf0f2f1a8 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/76934ce5c2b041f3942bf3fdf0f2f1a8 2023-03-30 16:54:11,984 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/76934ce5c2b041f3942bf3fdf0f2f1a8, entries=150, sequenceid=51, filesize=25.1 K 2023-03-30 16:54:11,985 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/fb0e085b67474bc48927096f115f44cc as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/fb0e085b67474bc48927096f115f44cc 2023-03-30 16:54:11,991 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/fb0e085b67474bc48927096f115f44cc, entries=150, sequenceid=51, filesize=11.6 K 2023-03-30 16:54:11,992 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/c6b6669a87be44ddac08518de61bb8f4 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/c6b6669a87be44ddac08518de61bb8f4 2023-03-30 16:54:11,996 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/c6b6669a87be44ddac08518de61bb8f4, entries=150, sequenceid=51, filesize=11.6 K 2023-03-30 16:54:11,997 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~375.17 KB/384176, currentSize=46.96 KB/48090 for 339a657579649f1a81384cc0c3fc4f47 in 1685ms, sequenceid=51, compaction requested=false 2023-03-30 16:54:11,997 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:11,997 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:11,997 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:11,997 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:11,998 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:11,998 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:11,998 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:11,999 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:11,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:11,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:12,000 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,000 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:12,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:12,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:12,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:12,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:12,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e0b678c[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:12,002 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:12,002 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,004 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,004 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:12,005 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:12,005 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:12,005 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,005 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:12,008 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,008 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:12,008 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:12,009 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:12,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:12,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:12,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:12,012 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:12,012 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:12,012 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:12,012 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:12,012 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:12,014 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,015 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:12,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,015 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:12,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:12,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:12,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:12,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:12,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,016 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:12,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:12,018 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:12,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:12,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:12,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:12,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,024 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:12,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:12,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:12,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,025 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,025 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,025 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,025 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:12,025 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:12,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:12,125 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:12,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:12,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:12,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:12,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:12,136 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:12,136 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:12,136 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:12,136 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:12,136 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,136 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,139 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:12,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:12,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:12,139 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,140 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,140 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:12,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:12,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:12,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:12,141 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:12,141 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:12,141 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:12,152 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:12,152 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:12,152 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:12,152 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:12,152 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:54:12,152 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:12,153 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:12,153 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:12,153 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:12,153 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:12,153 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:12,199 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303309715c196ca554a0cab5d1ee798438e18 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303309715c196ca554a0cab5d1ee798438e18 2023-03-30 16:54:12,215 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=61, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/a2a25068ca5f490ba9619168add2dfea 2023-03-30 16:54:12,243 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/0f16cfa4c0434839aac91d46d715ace9 2023-03-30 16:54:12,274 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/c7bc1edbca2c4137aaf812bef70755ad 2023-03-30 16:54:12,282 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/a2a25068ca5f490ba9619168add2dfea as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/a2a25068ca5f490ba9619168add2dfea 2023-03-30 16:54:12,290 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/a2a25068ca5f490ba9619168add2dfea, entries=150, sequenceid=61, filesize=25.1 K 2023-03-30 16:54:12,292 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/0f16cfa4c0434839aac91d46d715ace9 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/0f16cfa4c0434839aac91d46d715ace9 2023-03-30 16:54:12,302 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/0f16cfa4c0434839aac91d46d715ace9, entries=150, sequenceid=61, filesize=11.6 K 2023-03-30 16:54:12,309 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/c7bc1edbca2c4137aaf812bef70755ad as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/c7bc1edbca2c4137aaf812bef70755ad 2023-03-30 16:54:12,344 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/c7bc1edbca2c4137aaf812bef70755ad, entries=150, sequenceid=61, filesize=11.6 K 2023-03-30 16:54:12,345 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 339a657579649f1a81384cc0c3fc4f47 in 193ms, sequenceid=61, compaction requested=true 2023-03-30 16:54:12,345 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:12,345 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:12,345 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:12,345 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:12,345 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,345 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:12,346 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:12,347 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,347 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:12,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:12,348 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,348 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:12,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:12,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:12,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:12,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:12,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@529cd745[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:12,351 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:12,351 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,352 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,352 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:12,352 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:12,352 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:12,352 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,353 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:12,354 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:12,354 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:12,354 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:12,354 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:12,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:12,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:12,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:12,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:12,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:12,358 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:12,358 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:12,358 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:12,358 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:12,358 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:12,363 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,364 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:12,365 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:12,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:12,365 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:12,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:12,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:12,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:12,367 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:12,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,372 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,372 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,372 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,372 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:12,372 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:12,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:12,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:12,372 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:12,373 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:12,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:12,372 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:12,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:12,373 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:12,373 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,373 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,373 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:12,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:12,473 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:12,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:12,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:12,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:12,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:12,481 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:12,481 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:12,481 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:12,481 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:12,481 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,481 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,483 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:12,483 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:12,483 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,484 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:12,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:12,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:12,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:12,486 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:12,486 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:12,486 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:12,486 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:12,486 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:12,486 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:12,486 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:12,486 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:12,487 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:12,487 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:12,487 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:12,487 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,487 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:12,487 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:12,488 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,488 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,489 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,489 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:12,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:12,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:12,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:12,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:12,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:12,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d12a363[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:12,493 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:12,493 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,495 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,495 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:12,495 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:12,495 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:12,496 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,496 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:12,497 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,497 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:12,497 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:12,498 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:12,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:12,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:12,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:12,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:12,501 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:12,501 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:12,501 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:12,501 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:12,501 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:12,503 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,503 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,503 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:12,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:12,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:12,504 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,506 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:12,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,510 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,510 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,510 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,510 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,510 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,510 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,510 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:12,510 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:12,510 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:12,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:12,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:12,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:12,510 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:12,510 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:12,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:12,511 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:12,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:12,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:12,569 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:54:12,570 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:12,570 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:12,570 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:12,570 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:12,570 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:12,570 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:12,611 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:12,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:12,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:12,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680195252571, max timestamp=1680195252621Num uniques 859; 2023-03-30 16:54:12,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680195252571, max timestamp=1680195252621Num uniques 859; 2023-03-30 16:54:12,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-30 16:54:12,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680195252571, max timestamp=1680195252621Num uniques 859; 2023-03-30 16:54:12,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-30 16:54:12,624 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-30 16:54:12,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:12,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:12,629 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:12,629 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:12,629 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:12,629 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:12,630 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,630 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,633 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:12,633 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:12,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:12,633 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,633 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:12,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:12,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:12,636 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:12,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195312629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,636 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:12,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195312630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:12,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195312635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:12,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195312636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:12,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:12,641 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:12,641 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:12,641 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:12,656 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:12,656 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:12,656 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:12,656 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:12,656 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:12,656 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:12,656 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:12,656 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:12,656 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,656 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:12,656 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:12,658 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:12,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:12,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:12,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:12,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,660 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,660 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,660 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:12,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:12,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:12,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@731c4bc6[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:12,661 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:12,661 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,663 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,664 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:12,664 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:12,664 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:12,664 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,664 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:12,666 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:12,666 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,666 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:12,666 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:12,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:12,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:12,669 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:12,669 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:12,669 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:12,669 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:12,669 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:12,674 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,674 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,674 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:12,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:12,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:12,675 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:12,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:12,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:12,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:12,676 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:12,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,678 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,678 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,678 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,679 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,679 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:12,678 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:12,679 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:12,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:12,679 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:12,679 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:12,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:12,679 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:12,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:12,679 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,679 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:12,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:12,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:12,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:12,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195312738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:12,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195312738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:12,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195312742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:12,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195312742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,779 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:12,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:12,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:12,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:12,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:12,783 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:12,783 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:12,783 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:12,783 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:12,783 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,783 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,785 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:12,785 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:12,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:12,785 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,785 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:12,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:12,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:12,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:12,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:12,787 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:12,787 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:12,787 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:12,787 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:12,787 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:12,787 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:12,787 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:12,787 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:12,787 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:12,787 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:12,787 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:12,787 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,787 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:12,787 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:12,790 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,790 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:12,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:12,791 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,791 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:12,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:12,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:12,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:12,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:12,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16ce652e[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:12,792 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:12,793 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,794 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,794 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:12,794 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:12,794 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:12,794 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,794 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:12,796 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,796 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:12,796 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:12,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:12,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:12,796 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:12,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:12,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:12,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:12,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:12,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:12,799 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:12,799 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:12,799 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:12,799 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:12,799 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:12,801 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,801 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:12,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,801 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:12,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:12,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:12,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:12,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,802 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:12,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,803 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:12,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,806 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,806 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,806 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,806 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:12,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:12,806 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:12,807 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,807 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,807 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:12,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:12,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:12,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:12,811 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:12,811 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:12,911 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:12,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:12,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:12,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:12,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:12,915 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:12,915 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:12,915 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:12,915 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:12,915 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,915 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,917 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:12,917 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:12,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:12,917 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,917 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:12,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:12,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:12,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:12,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:12,918 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:12,918 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:12,918 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:12,918 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:12,918 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:12,919 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:12,919 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:12,919 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:12,919 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:12,919 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:12,919 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:12,919 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,919 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:12,919 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:12,920 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,920 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:12,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:12,921 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,921 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:12,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:12,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@225cbe4d[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:12,922 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:12,922 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,924 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,924 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:12,924 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:12,924 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:12,924 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,924 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:12,926 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,927 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:12,927 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,927 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:12,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:12,929 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:12,929 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:12,929 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:12,930 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:12,930 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:12,931 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,931 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,932 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:12,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,933 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:12,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,934 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:12,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,936 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,937 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:12,937 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:12,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:12,937 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:12,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:12,937 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:12,937 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:12,937 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:12,937 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:12,937 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,937 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,937 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:12,937 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:12,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:12,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:12,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195312944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:12,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195312944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:12,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195312945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:12,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:12,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195312945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,037 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:13,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:13,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:13,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:13,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:13,041 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:13,041 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:13,041 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:13,041 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:13,041 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,041 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,043 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:13,043 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:13,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:13,043 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,043 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:13,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:13,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:13,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:13,048 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:13,048 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:13,048 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:13,048 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:13,048 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:13,048 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:13,048 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:13,048 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:13,048 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:13,048 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:13,048 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:13,048 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,048 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:13,048 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:13,053 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,053 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,054 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,054 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:13,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4abd18de[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:13,055 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:13,055 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,056 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,056 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:13,056 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:13,056 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:13,057 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,057 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:13,058 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:13,058 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,058 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,058 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:13,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:13,061 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:13,061 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:13,061 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:13,061 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:13,061 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:13,062 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,062 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,063 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,063 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,065 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330771ab84c6f764b27bb4b346d4723ed90 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330771ab84c6f764b27bb4b346d4723ed90 2023-03-30 16:54:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,065 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,072 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,072 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:13,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:13,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:13,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:13,072 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:13,072 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:13,072 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,072 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:13,072 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:13,072 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,072 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,072 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,072 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,072 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:13,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:13,172 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:13,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:13,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:13,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:13,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:13,176 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:13,176 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:13,176 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:13,176 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:13,177 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,177 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,179 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:13,179 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:13,179 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,179 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:13,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:13,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:13,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:13,180 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:13,180 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:13,180 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:13,181 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:13,181 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:13,181 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:13,181 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:13,181 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:13,181 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:13,181 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:13,181 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:13,181 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,181 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:13,181 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:13,183 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,183 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,183 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,183 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:13,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:13,184 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:13,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bee2834[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:13,184 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,186 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,186 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:13,186 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:13,186 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:13,186 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,186 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:13,188 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,188 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,188 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:13,188 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:13,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:13,190 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:13,190 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:13,190 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:13,190 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:13,190 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:13,192 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,192 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:13,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,192 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:13,193 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:13,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,194 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,196 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:13,196 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:13,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:13,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:13,196 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:13,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:13,196 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:13,196 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,197 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,197 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,197 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,197 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,197 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,197 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:13,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:13,197 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:13,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:13,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195313247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:13,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195313247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:13,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195313249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:13,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195313249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,297 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:13,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:13,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:13,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:13,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:13,300 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:13,301 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:13,301 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:13,301 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:13,301 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,301 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,302 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,302 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:13,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:13,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:13,303 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,303 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:13,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:13,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:13,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:13,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:13,304 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:13,304 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:13,304 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:13,305 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:13,305 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:13,305 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:13,305 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:13,305 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:13,305 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:13,305 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:13,305 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:13,305 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,305 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:13,305 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:13,308 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,308 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,309 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,309 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:13,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:13,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e6cd172[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:13,310 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:13,310 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,311 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,311 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,311 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:13,311 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:13,311 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:13,311 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:13,313 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,313 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:13,313 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:13,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,313 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:13,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:13,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:13,315 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:13,315 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:13,315 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:13,315 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:13,315 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:13,317 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,317 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:13,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,317 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,318 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,319 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,322 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,322 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,322 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,322 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,322 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,322 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:13,323 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:13,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:13,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:13,322 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:13,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:13,323 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:13,323 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,323 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:13,323 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:13,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:13,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:13,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:13,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195313407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,413 DEBUG [Thread-733] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:54:13,423 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:13,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:13,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:13,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:13,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:13,426 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:13,427 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:13,427 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:13,427 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:13,427 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,427 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,428 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:13,428 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:13,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:13,429 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,429 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:13,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:13,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:13,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:13,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:13,430 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:13,430 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:13,430 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:13,432 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:13,432 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:13,432 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:13,432 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:13,432 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:13,432 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:13,432 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:13,432 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:13,432 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,432 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:13,432 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:13,434 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,434 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,434 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:13,435 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:13,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7985ae24[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:13,437 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:13,437 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,438 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,438 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,438 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:13,439 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:13,439 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:13,439 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:13,441 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,441 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:13,441 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:13,441 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:13,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:13,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:13,444 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:13,444 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:13,444 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:13,444 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:13,444 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:13,447 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,447 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:13,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,448 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:13,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,449 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,454 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,458 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,458 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:13,458 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:13,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:13,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:13,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:13,458 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:13,458 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,458 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,458 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,458 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,458 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,459 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:13,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:13,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:13,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:13,459 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:13,460 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:13,473 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=72, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/6a8e194d1d6c49e5a62dd0a6f0cfb919 2023-03-30 16:54:13,560 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:13,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:13,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:13,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:13,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:13,563 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:13,563 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:13,563 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:13,563 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:13,564 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,564 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,568 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,568 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:13,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:13,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:13,568 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,569 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:13,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:13,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:13,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:13,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:13,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:13,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:13,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:13,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:13,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:13,570 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:13,570 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:13,570 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:13,570 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:13,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:13,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:13,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,571 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:13,571 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:13,572 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,572 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,573 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,573 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:13,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:13,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b5c3222[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:13,574 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:13,574 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,575 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,576 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,576 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:13,576 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:13,576 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:13,576 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:13,578 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:13,578 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:13,578 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,578 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:13,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:13,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:13,581 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:13,581 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:13,581 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:13,581 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:13,581 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:13,582 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,582 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:13,582 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,583 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,584 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,586 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:13,586 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:13,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:13,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:13,586 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:13,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:13,587 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:13,586 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,587 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:13,587 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,587 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:13,587 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,587 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,587 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:13,587 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,687 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:13,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:13,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:13,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:13,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:13,691 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:13,691 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:13,691 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:13,691 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:13,692 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,692 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,693 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,693 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:13,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:13,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:13,693 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,693 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:13,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:13,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:13,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:13,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:13,695 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:13,695 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:13,695 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:13,695 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:13,695 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:13,695 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:13,695 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:13,695 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:13,695 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:13,695 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:13,695 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:13,695 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,695 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:13,695 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:13,698 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,698 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,698 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,698 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:13,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:13,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71d90ef5[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:13,700 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:13,700 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,701 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,701 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:13,701 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:13,701 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:13,701 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,701 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:13,703 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:13,703 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:13,703 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,703 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:13,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:13,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:13,706 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:13,706 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:13,706 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:13,706 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:13,706 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:13,708 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,708 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,708 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:13,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:13,709 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,710 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,714 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:13,714 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,714 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:13,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:13,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:13,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:13,714 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,714 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:13,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:13,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:13,714 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,714 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:13,714 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:13,714 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,714 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,714 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,714 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:13,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:13,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:13,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195313749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:13,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195313750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:13,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195313751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:13,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195313752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,814 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:13,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:13,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:13,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:13,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:13,818 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:13,818 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:13,818 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:13,818 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:13,819 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,819 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,820 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,820 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:13,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:13,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:13,820 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,820 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:13,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:13,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:13,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:13,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:13,822 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:13,822 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:13,822 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:13,822 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:13,822 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:13,822 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:13,822 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:13,822 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:13,822 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:13,822 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:13,822 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:13,823 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,823 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:13,823 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:13,824 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,824 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,824 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,825 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:13,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:13,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2966b6fd[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:13,826 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:13,826 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,828 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,828 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:13,828 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:13,828 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:13,828 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,828 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:13,830 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,831 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:13,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,831 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:13,831 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:13,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:13,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:13,832 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:13,832 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:13,832 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:13,832 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:13,832 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:13,834 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,834 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,835 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:13,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,835 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:13,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:13,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,837 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,840 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,840 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,840 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:13,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:13,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:13,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:13,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:13,840 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:13,840 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:13,840 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:13,840 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:13,840 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:13,840 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,840 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,840 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,840 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:13,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:13,885 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=72 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/9b57e36df4384beda8843ff2f511b351 2023-03-30 16:54:13,940 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:13,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:13,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:13,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:13,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:13,944 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:13,944 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:13,944 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:13,944 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:13,944 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,944 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,947 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:13,947 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:13,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:13,947 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,947 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:13,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:13,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:13,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:13,949 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:13,949 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:13,949 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:13,949 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:13,949 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:13,949 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:13,949 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:13,949 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:13,949 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:13,950 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:13,950 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:13,950 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,950 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:13,950 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:13,954 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,954 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,954 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,954 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:13,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc6d97b[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:13,956 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:13,956 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,957 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,957 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,957 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:13,957 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:13,957 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:13,958 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:13,959 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,959 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:13,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,959 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:13,959 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:13,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:13,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:13,966 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:13,966 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:13,966 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:13,966 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:13,966 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:13,969 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,969 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:13,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:13,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,969 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:13,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:13,970 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:13,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:13,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:13,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:13,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,971 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:13,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:13,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,973 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,973 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:13,973 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:13,973 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:13,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:13,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:13,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:13,973 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:13,973 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:13,974 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:13,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:13,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:13,974 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:13,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:13,974 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:13,974 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,974 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:13,974 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,074 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:14,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:14,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:14,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:14,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:14,077 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:14,077 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:14,077 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:14,078 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:14,078 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,078 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:14,079 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,079 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:14,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:14,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:14,080 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,080 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:14,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:14,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:14,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:14,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:14,081 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:14,081 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:14,081 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:14,081 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:14,081 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:14,081 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:14,081 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:14,081 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:14,081 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:14,081 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:14,081 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:14,081 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,082 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:14,082 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:14,083 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:14,083 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:14,084 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:14,084 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:14,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:14,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:14,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:14,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@473129c3[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:14,086 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:14,086 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:14,088 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:14,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:14,088 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:14,088 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:14,088 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:14,091 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:14,091 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,091 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:14,091 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:14,091 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:14,091 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:14,091 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:14,091 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:14,092 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:14,092 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:14,092 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:14,093 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,093 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:14,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,093 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:14,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:14,094 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:14,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:14,095 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:14,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:14,098 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,098 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:14,098 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:14,098 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,098 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:14,098 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:14,098 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:14,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:14,098 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:14,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:14,098 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:14,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:14,098 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:14,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:14,098 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:14,099 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:14,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:14,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:14,199 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:14,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:14,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:14,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:14,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:14,202 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:14,202 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:14,202 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:14,202 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:14,203 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,203 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:14,204 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,204 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:14,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:14,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:14,205 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,205 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:14,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:14,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:14,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:14,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:14,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:14,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:14,206 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:14,206 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:14,206 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:14,206 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:14,206 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:14,206 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:14,206 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:14,206 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:14,206 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:14,206 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:14,206 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:14,207 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,207 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:14,207 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:14,208 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,208 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:14,208 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:14,208 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:14,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:14,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ea63976[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:14,210 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:14,210 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:14,211 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:14,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:14,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:14,211 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:14,211 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:14,211 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:14,211 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,211 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:14,212 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:14,212 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,212 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:14,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:14,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:14,213 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:14,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:14,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:14,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:14,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:14,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:14,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:14,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:14,215 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:14,215 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:14,215 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:14,215 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:14,215 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:14,217 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,217 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:14,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:14,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,217 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:14,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:14,218 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:14,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:14,219 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:14,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:14,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,221 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,221 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:14,221 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:14,221 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:14,221 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:14,221 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:14,221 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:14,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:14,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:14,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:14,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:14,221 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:14,222 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:14,222 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:14,222 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:14,222 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:14,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:14,298 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=72 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/b7de9f2a279c4c6cbf1a04b011a9dc4e 2023-03-30 16:54:14,303 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/6a8e194d1d6c49e5a62dd0a6f0cfb919 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/6a8e194d1d6c49e5a62dd0a6f0cfb919 2023-03-30 16:54:14,307 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/6a8e194d1d6c49e5a62dd0a6f0cfb919, entries=350, sequenceid=72, filesize=52.3 K 2023-03-30 16:54:14,308 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/9b57e36df4384beda8843ff2f511b351 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/9b57e36df4384beda8843ff2f511b351 2023-03-30 16:54:14,312 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/9b57e36df4384beda8843ff2f511b351, entries=100, sequenceid=72, filesize=9.3 K 2023-03-30 16:54:14,313 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/b7de9f2a279c4c6cbf1a04b011a9dc4e as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/b7de9f2a279c4c6cbf1a04b011a9dc4e 2023-03-30 16:54:14,318 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/b7de9f2a279c4c6cbf1a04b011a9dc4e, entries=100, sequenceid=72, filesize=9.3 K 2023-03-30 16:54:14,318 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=154.31 KB/158010 for 339a657579649f1a81384cc0c3fc4f47 in 1749ms, sequenceid=72, compaction requested=true 2023-03-30 16:54:14,319 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:14,319 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:14,319 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:54:14,319 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:54:14,319 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:14,319 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:14,321 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:54:14,321 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 130744 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:54:14,321 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 339a657579649f1a81384cc0c3fc4f47/B is initiating minor compaction (all files) 2023-03-30 16:54:14,321 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 339a657579649f1a81384cc0c3fc4f47/A is initiating minor compaction (all files) 2023-03-30 16:54:14,321 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 339a657579649f1a81384cc0c3fc4f47/B in TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:14,321 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 339a657579649f1a81384cc0c3fc4f47/A in TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:14,321 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/71f8120f7bc148c79ec130c304a1d4f5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/fb0e085b67474bc48927096f115f44cc, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/0f16cfa4c0434839aac91d46d715ace9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/9b57e36df4384beda8843ff2f511b351] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp, totalSize=44.1 K 2023-03-30 16:54:14,321 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/ff30f19580ae4a7db821c88ecd4dbf4f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/76934ce5c2b041f3942bf3fdf0f2f1a8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/a2a25068ca5f490ba9619168add2dfea, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/6a8e194d1d6c49e5a62dd0a6f0cfb919] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp, totalSize=127.7 K 2023-03-30 16:54:14,322 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:14,322 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 71f8120f7bc148c79ec130c304a1d4f5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680195249108 2023-03-30 16:54:14,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:14,323 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting ff30f19580ae4a7db821c88ecd4dbf4f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680195249108 2023-03-30 16:54:14,323 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting fb0e085b67474bc48927096f115f44cc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680195249197 2023-03-30 16:54:14,324 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 76934ce5c2b041f3942bf3fdf0f2f1a8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680195249197 2023-03-30 16:54:14,324 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 0f16cfa4c0434839aac91d46d715ace9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680195250366 2023-03-30 16:54:14,324 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting a2a25068ca5f490ba9619168add2dfea, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680195250366 2023-03-30 16:54:14,324 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 9b57e36df4384beda8843ff2f511b351, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1680195252567 2023-03-30 16:54:14,325 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 6a8e194d1d6c49e5a62dd0a6f0cfb919, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1680195252523 2023-03-30 16:54:14,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:14,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:14,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:14,327 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:14,327 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:14,327 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:14,327 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:14,327 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,327 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:14,328 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:14,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:14,329 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:14,329 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:14,329 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:14,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:14,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:14,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:14,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:14,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:14,331 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:14,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:14,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:14,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:14,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:14,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:14,331 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:14,331 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:14,331 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=154.31 KB heapSize=379.13 KB 2023-03-30 16:54:14,332 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:14,332 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:14,332 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:14,332 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:14,332 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:14,332 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:14,337 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 339a657579649f1a81384cc0c3fc4f47#B#compaction#105 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:14,350 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 339a657579649f1a81384cc0c3fc4f47#A#compaction#106 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:14,395 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/b4d928bee68446eeaf32612219e0e5a1 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/b4d928bee68446eeaf32612219e0e5a1 2023-03-30 16:54:14,400 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 339a657579649f1a81384cc0c3fc4f47/A of 339a657579649f1a81384cc0c3fc4f47 into b4d928bee68446eeaf32612219e0e5a1(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:14,400 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:14,400 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., storeName=339a657579649f1a81384cc0c3fc4f47/A, priority=12, startTime=1680195254319; duration=0sec 2023-03-30 16:54:14,400 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:14,400 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:54:14,402 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:54:14,402 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 339a657579649f1a81384cc0c3fc4f47/C is initiating minor compaction (all files) 2023-03-30 16:54:14,402 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 339a657579649f1a81384cc0c3fc4f47/C in TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:14,402 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/6f346196a2a440daaaf88f8981701228, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/c6b6669a87be44ddac08518de61bb8f4, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/c7bc1edbca2c4137aaf812bef70755ad, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/b7de9f2a279c4c6cbf1a04b011a9dc4e] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp, totalSize=44.1 K 2023-03-30 16:54:14,403 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 6f346196a2a440daaaf88f8981701228, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680195249108 2023-03-30 16:54:14,403 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting c6b6669a87be44ddac08518de61bb8f4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680195249197 2023-03-30 16:54:14,404 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting c7bc1edbca2c4137aaf812bef70755ad, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680195250366 2023-03-30 16:54:14,404 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting b7de9f2a279c4c6cbf1a04b011a9dc4e, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1680195252567 2023-03-30 16:54:14,419 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 339a657579649f1a81384cc0c3fc4f47#C#compaction#108 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:14,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:14,760 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:14,778 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/784950d216cf4c699d0cefb154f43771 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/784950d216cf4c699d0cefb154f43771 2023-03-30 16:54:14,783 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303307753d60fa1974a7dbcb7ccb3dcbf69ef to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303307753d60fa1974a7dbcb7ccb3dcbf69ef 2023-03-30 16:54:14,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:14,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195314780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:14,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195314781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:14,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195314781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:14,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195314783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,807 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 339a657579649f1a81384cc0c3fc4f47/B of 339a657579649f1a81384cc0c3fc4f47 into 784950d216cf4c699d0cefb154f43771(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:14,807 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:14,807 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., storeName=339a657579649f1a81384cc0c3fc4f47/B, priority=12, startTime=1680195254319; duration=0sec 2023-03-30 16:54:14,808 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:14,821 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/832f7695d9b14441987f31e3dc58e248 2023-03-30 16:54:14,834 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/b7add7088a8448ae924fa779ffea0867 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/b7add7088a8448ae924fa779ffea0867 2023-03-30 16:54:14,840 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 339a657579649f1a81384cc0c3fc4f47/C of 339a657579649f1a81384cc0c3fc4f47 into b7add7088a8448ae924fa779ffea0867(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:14,840 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:14,840 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., storeName=339a657579649f1a81384cc0c3fc4f47/C, priority=12, startTime=1680195254319; duration=0sec 2023-03-30 16:54:14,840 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:14,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:14,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195314889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:14,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195314889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:14,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195314890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:14,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:14,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195314890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:15,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195315093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:15,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195315093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:15,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195315093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:15,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195315094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,253 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/21266e102cb34a12a261ca3d26d192ad 2023-03-30 16:54:15,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:15,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195315394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:15,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195315397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:15,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195315397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:15,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195315399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,664 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/f9b957a3755041e0beae12c143bdcadd 2023-03-30 16:54:15,669 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/832f7695d9b14441987f31e3dc58e248 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/832f7695d9b14441987f31e3dc58e248 2023-03-30 16:54:15,675 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/832f7695d9b14441987f31e3dc58e248, entries=150, sequenceid=98, filesize=25.1 K 2023-03-30 16:54:15,676 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/21266e102cb34a12a261ca3d26d192ad as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/21266e102cb34a12a261ca3d26d192ad 2023-03-30 16:54:15,681 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/21266e102cb34a12a261ca3d26d192ad, entries=150, sequenceid=98, filesize=11.6 K 2023-03-30 16:54:15,682 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/f9b957a3755041e0beae12c143bdcadd as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/f9b957a3755041e0beae12c143bdcadd 2023-03-30 16:54:15,706 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/f9b957a3755041e0beae12c143bdcadd, entries=150, sequenceid=98, filesize=11.6 K 2023-03-30 16:54:15,707 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~379.08 KB/388176, currentSize=46.96 KB/48090 for 339a657579649f1a81384cc0c3fc4f47 in 1376ms, sequenceid=98, compaction requested=false 2023-03-30 16:54:15,707 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:15,707 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:15,707 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:15,707 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:15,707 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,707 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:15,707 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:15,709 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,709 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:15,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:15,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:15,710 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:15,710 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:15,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:15,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:15,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:15,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:15,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:15,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62eba901[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:15,712 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:15,712 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:15,713 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:15,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:15,713 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:15,713 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:15,713 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:15,714 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,714 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:15,719 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:15,720 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:15,720 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:15,720 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:15,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:15,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:15,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:15,722 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:15,722 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:15,722 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:15,722 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:15,722 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:15,724 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:15,725 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:15,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:15,725 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:15,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:15,725 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:15,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:15,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:15,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:15,730 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:15,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:15,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:15,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:15,733 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:15,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:15,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:15,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:15,733 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:15,733 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:15,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:15,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:15,734 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:15,734 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:15,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:15,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:15,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:15,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:15,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:15,833 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:15,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:15,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:15,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:15,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:15,839 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:15,839 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:15,839 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:15,840 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:15,840 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:15,840 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:15,842 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:15,842 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:15,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:15,843 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,843 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:15,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:15,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:15,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:15,848 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:15,848 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:15,848 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:15,850 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:15,851 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:15,851 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:15,851 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:15,851 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:54:15,851 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:15,851 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:15,851 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:15,851 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:15,852 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:15,852 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:15,886 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303303b7713cb9b324f62980f0839e3eca9fa to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303b7713cb9b324f62980f0839e3eca9fa 2023-03-30 16:54:15,900 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:15,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:15,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:15,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195315941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:15,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195315944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680195255934, max timestamp=1680195255942Num uniques 859; 2023-03-30 16:54:15,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-30 16:54:15,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680195255934, max timestamp=1680195255942Num uniques 859; 2023-03-30 16:54:15,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680195255934, max timestamp=1680195255942Num uniques 859; 2023-03-30 16:54:15,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-30 16:54:15,945 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-30 16:54:15,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:15,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195315966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:15,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:15,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195315967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:16,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:16,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195316046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:16,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:16,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195316047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:16,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:16,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195316068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:16,072 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:16,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195316070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:16,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:16,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195316252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:16,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:16,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195316252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:16,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:16,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195316271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:16,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:16,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195316274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:16,294 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=111, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/089e79cdb1894cfbafa4dd3fbe323620 2023-03-30 16:54:16,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:16,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195316557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:16,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:16,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195316558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:16,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:16,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195316579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:16,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:16,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195316580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:16,704 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/ecfe805bf3224321ad9736f8e45cabd2 2023-03-30 16:54:17,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:17,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195317060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:17,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195317063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:17,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195317080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:17,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195317085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,119 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/bac88a6aeb9244af834887b421ed8bdb 2023-03-30 16:54:17,125 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/089e79cdb1894cfbafa4dd3fbe323620 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/089e79cdb1894cfbafa4dd3fbe323620 2023-03-30 16:54:17,135 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/089e79cdb1894cfbafa4dd3fbe323620, entries=150, sequenceid=111, filesize=25.1 K 2023-03-30 16:54:17,136 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/ecfe805bf3224321ad9736f8e45cabd2 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/ecfe805bf3224321ad9736f8e45cabd2 2023-03-30 16:54:17,140 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/ecfe805bf3224321ad9736f8e45cabd2, entries=150, sequenceid=111, filesize=11.6 K 2023-03-30 16:54:17,142 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/bac88a6aeb9244af834887b421ed8bdb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/bac88a6aeb9244af834887b421ed8bdb 2023-03-30 16:54:17,147 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/bac88a6aeb9244af834887b421ed8bdb, entries=150, sequenceid=111, filesize=11.6 K 2023-03-30 16:54:17,148 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 339a657579649f1a81384cc0c3fc4f47 in 1297ms, sequenceid=111, compaction requested=true 2023-03-30 16:54:17,148 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:17,148 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:17,149 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:17,149 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:17,149 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,149 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:17,149 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:17,150 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:17,150 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:17,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:17,151 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:17,151 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:17,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:17,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:17,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:17,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:17,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:17,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@292da926[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:17,153 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:17,153 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:17,155 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:17,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:17,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:17,155 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:17,155 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:17,155 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:17,155 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,155 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:17,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,157 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:17,157 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:17,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:17,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:17,157 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:17,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:17,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:17,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:17,161 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:17,161 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:17,161 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:17,161 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:17,161 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:17,162 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:17,162 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:17,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:17,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:17,162 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:17,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:17,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:17,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:17,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:17,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:17,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:17,164 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:17,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:17,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:17,166 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:17,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:17,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:17,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,169 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,169 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:17,169 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:17,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:17,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:17,169 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:17,169 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,169 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:17,169 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:17,169 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:17,169 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:17,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:17,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:17,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:17,170 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:17,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:17,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:17,170 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:17,170 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:17,270 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:17,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:17,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:17,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:17,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:17,274 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:17,274 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:17,274 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:17,274 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:17,274 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:17,275 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:17,276 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:17,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:17,277 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:17,278 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,278 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:17,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:17,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:17,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:17,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:17,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:17,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:17,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:17,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:17,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:17,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:17,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:17,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:17,280 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:17,280 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:17,280 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=140.89 KB heapSize=386.16 KB 2023-03-30 16:54:17,281 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:17,281 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:17,281 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:17,281 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:17,281 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:17,281 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:17,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:17,435 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:17,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:17,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195317458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:17,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195317562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:17,696 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330da304815d38741de8c916ac40bb1f837 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330da304815d38741de8c916ac40bb1f837 2023-03-30 16:54:17,705 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=145, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/2f62792d80d24c479e3320d84d9a8b5f 2023-03-30 16:54:17,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:17,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195317765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:18,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195318069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:18,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195318069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:18,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195318072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:18,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195318099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:18,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195318099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,122 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/e52e1c84638e404fa151fa1bd04e2756 2023-03-30 16:54:18,536 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/f31872233b68450db1840a4023c9f2c6 2023-03-30 16:54:18,545 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/2f62792d80d24c479e3320d84d9a8b5f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/2f62792d80d24c479e3320d84d9a8b5f 2023-03-30 16:54:18,549 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/2f62792d80d24c479e3320d84d9a8b5f, entries=150, sequenceid=145, filesize=25.3 K 2023-03-30 16:54:18,550 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/e52e1c84638e404fa151fa1bd04e2756 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/e52e1c84638e404fa151fa1bd04e2756 2023-03-30 16:54:18,554 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/e52e1c84638e404fa151fa1bd04e2756, entries=150, sequenceid=145, filesize=11.8 K 2023-03-30 16:54:18,555 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/f31872233b68450db1840a4023c9f2c6 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/f31872233b68450db1840a4023c9f2c6 2023-03-30 16:54:18,559 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/f31872233b68450db1840a4023c9f2c6, entries=150, sequenceid=145, filesize=11.8 K 2023-03-30 16:54:18,560 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~386.11 KB/395376, currentSize=46.96 KB/48090 for 339a657579649f1a81384cc0c3fc4f47 in 1280ms, sequenceid=145, compaction requested=true 2023-03-30 16:54:18,560 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:18,560 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:18,562 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,562 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:18,562 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,562 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:18,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:18,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@671cf2f1[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:18,564 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:18,564 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,566 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,566 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:18,566 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:18,566 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:18,566 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,566 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:18,567 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:18,567 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,567 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:18,567 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:18,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:18,570 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:18,570 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:18,570 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:18,570 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:18,570 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:18,572 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,572 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,572 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:18,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,572 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:18,573 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:18,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,576 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:18,576 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:18,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:18,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:18,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:18,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:18,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:18,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:18,577 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 16:54:18,577 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:18,577 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:18,577 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:18,577 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:18,578 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:18,578 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:18,584 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,584 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:18,584 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:18,584 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,584 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,584 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,584 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,584 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:18,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:18,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:18,584 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:18,584 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:18,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:18,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195318640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680195258609, max timestamp=1680195258639Num uniques 859; 2023-03-30 16:54:18,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680195258609, max timestamp=1680195258639Num uniques 859; 2023-03-30 16:54:18,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680195258609, max timestamp=1680195258639Num uniques 859; 2023-03-30 16:54:18,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 16:54:18,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 16:54:18,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 16:54:18,685 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:18,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:18,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:18,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:18,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:18,688 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:18,688 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:18,688 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:18,688 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:18,688 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,689 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:18,691 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:18,691 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:18,691 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,692 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:18,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:18,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:18,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:18,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:18,693 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:18,693 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:18,693 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:18,693 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:18,693 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:18,693 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:18,693 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:18,693 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:18,693 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:18,693 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:18,693 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:18,693 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,693 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:18,693 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:18,695 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,695 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:18,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:18,695 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,695 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:18,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:18,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:18,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:18,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:18,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28be0aed[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:18,697 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:18,697 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,698 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,698 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:18,698 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,698 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:18,698 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:18,699 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:18,700 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:18,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:18,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:18,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:18,703 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:18,703 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:18,703 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:18,703 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:18,703 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:18,709 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,709 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,709 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:18,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:18,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:18,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:18,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:18,709 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,711 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,715 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,715 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:18,715 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:18,715 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:18,715 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:18,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:18,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:18,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:18,715 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:18,715 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:18,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:18,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:18,716 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:18,716 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,716 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:18,716 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,716 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:18,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195318780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,816 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:18,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:18,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:18,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:18,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:18,819 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:18,819 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:18,820 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:18,820 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:18,820 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,820 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:18,821 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:18,821 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:18,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:18,822 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,822 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:18,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:18,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:18,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:18,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:18,823 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:18,823 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:18,823 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:18,823 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:18,823 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:18,823 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:18,823 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:18,823 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:18,823 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:18,823 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:18,823 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:18,823 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,823 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:18,823 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:18,825 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:18,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,825 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:18,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:18,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:18,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17220d5a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:18,827 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:18,827 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,829 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,829 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:18,829 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:18,829 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,830 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:18,829 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:18,831 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,831 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:18,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:18,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:18,831 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:18,831 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:18,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:18,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:18,833 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:18,834 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:18,834 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:18,834 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:18,834 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:18,835 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,835 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:18,835 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:18,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:18,836 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,837 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:18,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,839 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:18,839 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:18,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:18,839 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:18,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:18,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:18,840 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:18,840 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:18,840 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:18,840 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,840 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,840 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,840 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,840 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:18,840 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:18,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:18,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:18,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195318882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,940 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:18,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:18,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:18,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:18,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:18,944 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:18,944 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:18,944 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:18,944 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:18,944 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,944 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:18,946 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:18,947 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:18,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:18,947 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,947 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:18,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:18,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:18,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:18,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:18,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:18,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:18,948 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:18,948 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:18,948 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:18,948 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:18,948 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:18,948 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:18,948 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:18,949 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:18,949 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:18,949 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:18,949 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:18,949 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,949 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:18,949 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:18,950 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,950 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:18,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:18,951 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,951 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:18,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:18,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:18,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:18,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:18,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5382f201[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:18,952 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:18,952 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,954 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,954 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:18,954 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:18,954 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:18,959 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,959 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:18,960 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,960 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:18,960 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:18,960 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:18,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:18,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:18,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:18,963 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:18,963 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:18,963 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:18,963 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:18,963 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:18,968 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,968 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:18,968 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:18,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:18,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:18,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:18,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:18,969 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,970 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:18,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:18,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,972 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,972 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:18,972 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:18,972 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:18,972 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:18,972 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:18,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:18,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:18,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:18,972 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:18,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:18,972 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:18,973 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:18,973 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:18,973 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:18,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:18,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:18,973 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:18,992 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330a6c7a483853b47a4b9bb65dad9003d45 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330a6c7a483853b47a4b9bb65dad9003d45 2023-03-30 16:54:19,073 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:19,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:19,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:19,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:19,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:19,076 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:19,076 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:19,077 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:19,077 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:19,077 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,077 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,079 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:19,079 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:19,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:19,079 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,079 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:19,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:19,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:19,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:19,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:19,080 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:19,080 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:19,080 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:19,080 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:19,080 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:19,081 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:19,081 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:19,081 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:19,081 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:19,081 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:19,081 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:19,081 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,081 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:19,081 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:19,082 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,082 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:19,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:19,083 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,083 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:19,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:19,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:19,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:19,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:19,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@530919fa[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:19,084 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:19,085 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,087 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,087 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:19,087 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:19,087 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:19,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:19,087 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195319085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,088 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:19,089 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:19,089 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:19,089 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,089 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:19,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:19,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:19,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:19,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:19,091 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:19,091 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:19,091 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:19,091 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:19,091 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:19,093 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,093 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,093 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:19,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:19,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:19,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:19,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:19,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:19,094 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:19,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,095 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,098 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,098 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,099 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:19,099 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:19,099 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:19,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:19,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:19,099 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:19,099 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,100 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:19,100 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:19,100 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,100 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:19,100 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,200 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:19,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:19,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:19,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:19,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:19,203 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:19,204 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:19,204 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:19,204 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:19,204 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,204 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,205 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,206 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:19,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:19,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:19,206 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,206 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:19,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:19,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:19,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:19,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:19,207 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:19,207 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:19,207 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:19,207 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:19,207 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:19,207 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:19,207 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:19,207 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:19,207 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:19,207 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:19,207 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:19,207 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,207 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:19,207 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:19,209 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,209 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:19,209 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,209 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:19,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:19,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b388814[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:19,211 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:19,211 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,212 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,212 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:19,212 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:19,212 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:19,212 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,212 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:19,214 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:19,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:19,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:19,217 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:19,217 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:19,217 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:19,217 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:19,217 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:19,218 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,218 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,218 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:19,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:19,219 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,220 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,222 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:19,222 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:19,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:19,222 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:19,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:19,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:19,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:19,222 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,223 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,223 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:19,223 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:19,223 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,223 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:19,223 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,223 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:19,223 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:19,323 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:19,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:19,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:19,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:19,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:19,326 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:19,326 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:19,326 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:19,326 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:19,327 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,327 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,328 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:19,328 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:19,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:19,329 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,329 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:19,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:19,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:19,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:19,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:19,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:19,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:19,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:19,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:19,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:19,331 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:19,331 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:19,331 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:19,331 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:19,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:19,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:19,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:19,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:19,333 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:19,334 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:19,334 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,334 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:19,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:19,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c771cf4[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:19,335 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:19,335 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,336 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:19,337 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,337 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:19,338 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,338 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:19,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:19,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:19,338 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:19,338 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:19,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:19,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:19,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:19,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:19,343 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:19,343 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:19,343 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:19,343 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:19,343 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:19,346 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,346 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:19,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,346 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:19,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:19,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:19,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:19,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:19,347 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:19,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,349 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:19,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,354 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:19,354 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:19,354 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,354 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,354 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,354 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,354 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,354 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,354 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:19,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:19,358 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:19,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:19,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:19,359 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:19,359 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:19,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:19,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195319388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,398 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=156, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/978a909344ed474b9f6c32db063aad83 2023-03-30 16:54:19,414 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/2c6eccb49f2f46eea8e51fa4eecea7b7 2023-03-30 16:54:19,459 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:19,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:19,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:19,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:19,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:19,462 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:19,462 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:19,462 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:19,462 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:19,463 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,463 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,464 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,464 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:19,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:19,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:19,464 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,464 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:19,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:19,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:19,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:19,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:19,466 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:19,466 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:19,466 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:19,466 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:19,466 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:19,466 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:19,466 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:19,466 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:19,466 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:19,466 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:19,466 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:19,466 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,467 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:19,467 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:19,468 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:19,471 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:19,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48a1f581[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:19,471 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:19,471 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,471 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:19,471 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,473 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,474 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:19,474 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:19,474 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:19,475 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,475 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:19,475 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,475 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:19,475 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:19,475 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:19,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:19,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:19,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:19,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:19,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:19,478 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:19,478 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:19,478 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:19,478 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:19,478 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:19,480 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,480 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:19,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,480 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:19,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:19,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:19,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:19,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:19,482 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:19,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,493 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:19,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,499 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,499 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,499 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,499 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,499 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,499 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,499 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,499 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:19,499 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:19,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:19,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:19,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:19,499 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:19,500 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:19,500 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:19,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:19,600 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:19,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:19,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:19,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:19,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:19,603 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:19,603 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:19,603 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:19,603 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:19,603 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,603 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,605 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:19,605 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:19,605 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,605 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:19,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:19,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:19,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:19,606 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:19,606 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:19,606 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:19,607 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:19,607 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:19,607 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:19,607 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:19,607 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:19,607 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:19,607 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:19,607 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:19,607 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,607 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:19,607 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:19,608 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,608 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:19,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:19,609 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,609 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:19,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:19,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:19,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:19,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:19,610 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:19,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53fce3a6[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:19,610 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,612 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,613 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:19,613 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:19,613 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:19,613 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,613 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:19,614 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,614 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:19,614 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:19,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:19,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:19,614 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:19,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:19,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:19,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:19,616 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:19,616 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:19,616 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:19,616 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:19,617 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:19,618 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,618 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,618 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:19,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:19,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:19,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,618 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:19,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:19,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:19,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:19,620 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:19,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:19,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:19,622 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:19,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:19,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:19,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:19,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:19,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:19,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,622 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:19,623 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:19,623 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:19,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:19,723 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:19,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:19,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:19,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:19,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:19,726 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:19,726 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:19,726 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:19,726 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:19,727 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,727 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,729 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,729 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:19,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:19,730 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,730 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:19,731 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:19,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:19,731 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:19,731 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:19,731 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:19,731 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:19,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:19,733 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:19,733 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,733 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:19,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:19,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c4bb9e5[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:19,734 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:19,734 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,735 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,736 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:19,736 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:19,736 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:19,736 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,736 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:19,737 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,737 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:19,737 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:19,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,737 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:19,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:19,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:19,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:19,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:19,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:19,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:19,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:19,739 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:19,739 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:19,739 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:19,739 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:19,739 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:19,742 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,742 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:19,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:19,742 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:19,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,742 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:19,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,744 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,746 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,746 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,746 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:19,746 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:19,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:19,746 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:19,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:19,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:19,746 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:19,746 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,746 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,746 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:19,747 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:19,747 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:19,747 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:19,826 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/3e8ee5cebdd541f2b33fb0d901b98594 2023-03-30 16:54:19,831 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/978a909344ed474b9f6c32db063aad83 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/978a909344ed474b9f6c32db063aad83 2023-03-30 16:54:19,837 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/978a909344ed474b9f6c32db063aad83, entries=200, sequenceid=156, filesize=32.1 K 2023-03-30 16:54:19,838 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/2c6eccb49f2f46eea8e51fa4eecea7b7 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2c6eccb49f2f46eea8e51fa4eecea7b7 2023-03-30 16:54:19,843 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2c6eccb49f2f46eea8e51fa4eecea7b7, entries=150, sequenceid=156, filesize=11.8 K 2023-03-30 16:54:19,844 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/3e8ee5cebdd541f2b33fb0d901b98594 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/3e8ee5cebdd541f2b33fb0d901b98594 2023-03-30 16:54:19,847 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:19,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:19,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:19,850 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/3e8ee5cebdd541f2b33fb0d901b98594, entries=150, sequenceid=156, filesize=11.8 K 2023-03-30 16:54:19,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:19,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:19,851 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:19,851 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:19,851 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:19,851 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:19,851 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 339a657579649f1a81384cc0c3fc4f47 in 1274ms, sequenceid=156, compaction requested=true 2023-03-30 16:54:19,851 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:19,851 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:19,851 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 16:54:19,851 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 16:54:19,851 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:19,851 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:19,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,853 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:19,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:19,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:19,853 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,853 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 136158 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 16:54:19,854 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,854 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:19,854 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 339a657579649f1a81384cc0c3fc4f47/A is initiating minor compaction (all files) 2023-03-30 16:54:19,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:19,854 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 339a657579649f1a81384cc0c3fc4f47/A in TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:19,854 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59868 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 16:54:19,854 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/b4d928bee68446eeaf32612219e0e5a1, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/832f7695d9b14441987f31e3dc58e248, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/089e79cdb1894cfbafa4dd3fbe323620, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/2f62792d80d24c479e3320d84d9a8b5f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/978a909344ed474b9f6c32db063aad83] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp, totalSize=133.0 K 2023-03-30 16:54:19,854 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 339a657579649f1a81384cc0c3fc4f47/B is initiating minor compaction (all files) 2023-03-30 16:54:19,854 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 339a657579649f1a81384cc0c3fc4f47/B in TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:19,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:19,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:19,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:19,854 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/784950d216cf4c699d0cefb154f43771, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/21266e102cb34a12a261ca3d26d192ad, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/ecfe805bf3224321ad9736f8e45cabd2, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/e52e1c84638e404fa151fa1bd04e2756, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2c6eccb49f2f46eea8e51fa4eecea7b7] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp, totalSize=58.5 K 2023-03-30 16:54:19,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:19,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:19,855 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:19,855 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:19,855 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting b4d928bee68446eeaf32612219e0e5a1, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1680195250370 2023-03-30 16:54:19,855 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:19,855 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 784950d216cf4c699d0cefb154f43771, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1680195250370 2023-03-30 16:54:19,855 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:19,855 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:19,855 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:19,855 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:19,855 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 21266e102cb34a12a261ca3d26d192ad, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680195252623 2023-03-30 16:54:19,855 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-30 16:54:19,855 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 832f7695d9b14441987f31e3dc58e248, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680195252623 2023-03-30 16:54:19,856 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:19,856 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting ecfe805bf3224321ad9736f8e45cabd2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680195254758 2023-03-30 16:54:19,856 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:19,856 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:19,856 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:19,856 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:19,856 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:19,856 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting e52e1c84638e404fa151fa1bd04e2756, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680195255958 2023-03-30 16:54:19,856 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 089e79cdb1894cfbafa4dd3fbe323620, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680195254758 2023-03-30 16:54:19,857 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 2c6eccb49f2f46eea8e51fa4eecea7b7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1680195257439 2023-03-30 16:54:19,857 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 2f62792d80d24c479e3320d84d9a8b5f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680195255958 2023-03-30 16:54:19,858 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 978a909344ed474b9f6c32db063aad83, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1680195257439 2023-03-30 16:54:19,881 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 339a657579649f1a81384cc0c3fc4f47#B#compaction#122 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:19,894 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 339a657579649f1a81384cc0c3fc4f47#A#compaction#121 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:19,895 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033003bab6cdb1744793b8fe71c8e97857c4 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033003bab6cdb1744793b8fe71c8e97857c4 2023-03-30 16:54:19,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:19,900 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:19,908 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/0bc28f09f4ef4283bce5e1479420a847 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/0bc28f09f4ef4283bce5e1479420a847 2023-03-30 16:54:19,909 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=190, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/a7ae7ba1cbc0406db15c10412dabb987 2023-03-30 16:54:19,921 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 339a657579649f1a81384cc0c3fc4f47/B of 339a657579649f1a81384cc0c3fc4f47 into 0bc28f09f4ef4283bce5e1479420a847(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:19,921 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:19,921 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., storeName=339a657579649f1a81384cc0c3fc4f47/B, priority=11, startTime=1680195259851; duration=0sec 2023-03-30 16:54:19,921 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:19,922 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 16:54:19,923 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59868 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 16:54:19,923 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 339a657579649f1a81384cc0c3fc4f47/C is initiating minor compaction (all files) 2023-03-30 16:54:19,924 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 339a657579649f1a81384cc0c3fc4f47/C in TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:19,924 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/b7add7088a8448ae924fa779ffea0867, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/f9b957a3755041e0beae12c143bdcadd, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/bac88a6aeb9244af834887b421ed8bdb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/f31872233b68450db1840a4023c9f2c6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/3e8ee5cebdd541f2b33fb0d901b98594] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp, totalSize=58.5 K 2023-03-30 16:54:19,924 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting b7add7088a8448ae924fa779ffea0867, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1680195250370 2023-03-30 16:54:19,925 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting f9b957a3755041e0beae12c143bdcadd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680195252623 2023-03-30 16:54:19,925 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting bac88a6aeb9244af834887b421ed8bdb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680195254758 2023-03-30 16:54:19,925 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting f31872233b68450db1840a4023c9f2c6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1680195255958 2023-03-30 16:54:19,926 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 3e8ee5cebdd541f2b33fb0d901b98594, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1680195257439 2023-03-30 16:54:19,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:19,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195319930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:19,950 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 339a657579649f1a81384cc0c3fc4f47#C#compaction#124 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:19,970 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/c926aba046134185afa3874591258d2d 2023-03-30 16:54:19,979 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/4ef58e7a82cb480393f818f45c47f71b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/4ef58e7a82cb480393f818f45c47f71b 2023-03-30 16:54:19,984 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/dd2ee5edb19b439bba4b172298117601 2023-03-30 16:54:19,990 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 339a657579649f1a81384cc0c3fc4f47/C of 339a657579649f1a81384cc0c3fc4f47 into 4ef58e7a82cb480393f818f45c47f71b(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:19,990 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:19,990 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., storeName=339a657579649f1a81384cc0c3fc4f47/C, priority=11, startTime=1680195259851; duration=0sec 2023-03-30 16:54:19,990 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:19,993 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/a7ae7ba1cbc0406db15c10412dabb987 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/a7ae7ba1cbc0406db15c10412dabb987 2023-03-30 16:54:19,998 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/a7ae7ba1cbc0406db15c10412dabb987, entries=150, sequenceid=190, filesize=25.3 K 2023-03-30 16:54:19,998 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/c926aba046134185afa3874591258d2d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/c926aba046134185afa3874591258d2d 2023-03-30 16:54:20,003 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/c926aba046134185afa3874591258d2d, entries=150, sequenceid=190, filesize=11.8 K 2023-03-30 16:54:20,004 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/dd2ee5edb19b439bba4b172298117601 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/dd2ee5edb19b439bba4b172298117601 2023-03-30 16:54:20,008 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/dd2ee5edb19b439bba4b172298117601, entries=150, sequenceid=190, filesize=11.8 K 2023-03-30 16:54:20,008 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 339a657579649f1a81384cc0c3fc4f47 in 153ms, sequenceid=190, compaction requested=false 2023-03-30 16:54:20,008 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:20,008 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:20,009 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:20,009 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:20,009 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,015 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:20,015 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:20,017 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,017 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,017 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,017 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:20,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:20,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@286f8624[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:20,019 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:20,019 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,021 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,021 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:20,021 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:20,021 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:20,021 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,021 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:20,023 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,023 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:20,023 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:20,023 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:20,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:20,025 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:20,025 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:20,025 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:20,026 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:20,026 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:20,027 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,027 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,027 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:20,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:20,028 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,031 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,033 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,033 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,033 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,033 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:20,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:20,033 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,033 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,033 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,033 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:20,033 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:20,033 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:20,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:20,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:20,033 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:20,033 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:20,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:20,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:20,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:20,039 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-30 16:54:20,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:20,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:20,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:20,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:20,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:20,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:20,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:20,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195320093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:20,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195320093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,097 DEBUG [Thread-735] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:54:20,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:20,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195320095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:20,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195320114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,118 DEBUG [Thread-739] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:54:20,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:20,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195320117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,120 DEBUG [Thread-741] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:54:20,134 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:20,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:20,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:20,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:20,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:20,137 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:20,137 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:20,137 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:20,137 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:20,137 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,137 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,139 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:20,139 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:20,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:20,140 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,140 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:20,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:20,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:20,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:20,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:20,141 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:20,141 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:20,141 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:20,141 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:20,141 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:20,141 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:20,141 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:20,141 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:20,141 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:20,141 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:20,141 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:20,142 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,142 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:20,142 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:20,145 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,145 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:20,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:20,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fa57355[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:20,147 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:20,147 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,148 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,149 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:20,149 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:20,149 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:20,149 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,149 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:20,153 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,153 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:20,153 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:20,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,153 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:20,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:20,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:20,155 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:20,156 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:20,156 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:20,156 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:20,156 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:20,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:20,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:20,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,158 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,159 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,162 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,162 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,162 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,162 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:20,162 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:20,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:20,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:20,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:20,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:20,162 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:20,162 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:20,162 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:20,162 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,162 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,163 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,163 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:20,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:20,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:20,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195320196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:20,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195320200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,263 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:20,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:20,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:20,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:20,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:20,267 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:20,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:20,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:20,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:20,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,268 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:20,268 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:20,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:20,269 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,269 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:20,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:20,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:20,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:20,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:20,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:20,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:20,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:20,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:20,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:20,270 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:20,270 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:20,270 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:20,270 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:20,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:20,271 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:20,271 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,271 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:20,271 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:20,273 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,273 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,273 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,273 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:20,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:20,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bf05cd4[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:20,275 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:20,275 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,277 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,277 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,277 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:20,277 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:20,277 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:20,277 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:20,278 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:20,278 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:20,278 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,278 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:20,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:20,281 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:20,281 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:20,281 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:20,281 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:20,281 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:20,282 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,282 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,282 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:20,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:20,283 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,284 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,287 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,287 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:20,287 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:20,287 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:20,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:20,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:20,287 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:20,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:20,288 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:20,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:20,287 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,288 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:20,288 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,288 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,288 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,288 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,388 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:20,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:20,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:20,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:20,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:20,391 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:20,391 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:20,391 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:20,391 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:20,392 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,392 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,393 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,393 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:20,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:20,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:20,393 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,393 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:20,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:20,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:20,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:20,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:20,394 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:20,394 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:20,394 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:20,395 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:20,395 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:20,395 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:20,395 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:20,395 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:20,395 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:20,395 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:20,395 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:20,395 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,395 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:20,395 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:20,396 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,396 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,397 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,397 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:20,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:20,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:20,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195320398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ab542de[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:20,398 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:20,398 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,403 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:20,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:20,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:20,404 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,404 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:20,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:20,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195320402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,405 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,405 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:20,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,405 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:20,405 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:20,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:20,408 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:20,408 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:20,408 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:20,408 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:20,408 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:20,409 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,409 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,409 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:20,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,410 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,411 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,413 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:20,413 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,414 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:20,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:20,414 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:20,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:20,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:20,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:20,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:20,414 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,414 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,414 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,414 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,414 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:20,414 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:20,414 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,414 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:20,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:20,453 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303306a4b7bd077584e909a915e2596314330 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303306a4b7bd077584e909a915e2596314330 2023-03-30 16:54:20,514 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:20,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:20,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:20,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:20,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:20,517 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:20,517 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:20,517 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:20,517 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:20,518 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,518 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,519 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,519 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:20,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:20,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:20,519 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,519 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:20,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:20,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:20,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:20,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:20,520 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:20,520 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:20,520 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:20,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:20,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:20,521 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:20,521 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:20,521 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:20,521 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:20,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:20,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:20,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:20,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:20,523 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,523 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,523 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,523 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:20,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:20,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ebfa90a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:20,525 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,526 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:20,526 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:20,526 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:20,526 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,526 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:20,531 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,531 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:20,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,531 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:20,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,531 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:20,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:20,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:20,533 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:20,533 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:20,533 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:20,533 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:20,533 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:20,535 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,535 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,535 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,535 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:20,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,537 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,539 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,539 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,539 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:20,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:20,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:20,539 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,539 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:20,539 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:20,539 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,539 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,539 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,539 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:20,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:20,539 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:20,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:20,539 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:20,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:20,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:20,640 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:20,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:20,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:20,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:20,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:20,643 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:20,643 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:20,643 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:20,643 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:20,643 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,643 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,645 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:20,645 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:20,645 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,645 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:20,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:20,646 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:20,646 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:20,646 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:20,646 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:20,646 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:20,646 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:20,646 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:20,647 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:20,647 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:20,647 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:20,647 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:20,647 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:20,647 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:20,647 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,647 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:20,647 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:20,649 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,649 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,650 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,650 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:20,651 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:20,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a445453[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:20,651 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,652 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,652 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:20,652 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:20,652 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,652 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:20,652 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:20,654 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:20,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:20,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:20,656 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:20,656 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:20,657 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:20,657 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:20,657 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:20,658 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,658 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,658 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:20,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:20,659 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,660 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,664 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,664 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,664 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,664 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,664 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:20,664 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:20,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:20,664 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:20,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:20,664 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:20,664 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:20,664 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:20,664 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:20,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:20,664 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:20,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:20,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:20,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195320699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:20,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195320706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,746 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/6e7c475539914de38cc5d05ac28927cc as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/6e7c475539914de38cc5d05ac28927cc 2023-03-30 16:54:20,751 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 339a657579649f1a81384cc0c3fc4f47/A of 339a657579649f1a81384cc0c3fc4f47 into 6e7c475539914de38cc5d05ac28927cc(size=25.6 K), total size for store is 50.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:20,751 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:20,751 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., storeName=339a657579649f1a81384cc0c3fc4f47/A, priority=11, startTime=1680195259851; duration=0sec 2023-03-30 16:54:20,751 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:20,764 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:20,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:20,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:20,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:20,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:20,767 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:20,767 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:20,767 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:20,767 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:20,768 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,768 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,769 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,769 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:20,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:20,770 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,770 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:20,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:20,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:20,771 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:20,771 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:20,771 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:20,771 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:20,771 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:20,771 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:20,771 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:20,771 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:20,771 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:20,771 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:20,771 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:20,771 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,771 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:20,771 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:20,773 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,773 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,773 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,773 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:20,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@538bbf33[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:20,775 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:20,775 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,776 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,776 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:20,776 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,776 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:20,776 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:20,776 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:20,781 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:20,781 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:20,781 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,781 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:20,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:20,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:20,784 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:20,784 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:20,784 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:20,784 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:20,784 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:20,785 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,785 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:20,785 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,786 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,787 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,789 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,789 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:20,789 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:20,790 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:20,789 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:20,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:20,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:20,789 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,790 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,790 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,790 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,790 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,790 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:20,790 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:20,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:20,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:20,859 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=204, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/ca52c21756634d7480b8afd80818ce07 2023-03-30 16:54:20,870 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/efa49f544bda4991beca3b7e3066fe4d 2023-03-30 16:54:20,880 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/414b3a60ba9a4f8595599dba6314ddf0 2023-03-30 16:54:20,884 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/ca52c21756634d7480b8afd80818ce07 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/ca52c21756634d7480b8afd80818ce07 2023-03-30 16:54:20,888 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/ca52c21756634d7480b8afd80818ce07, entries=150, sequenceid=204, filesize=25.3 K 2023-03-30 16:54:20,889 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/efa49f544bda4991beca3b7e3066fe4d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/efa49f544bda4991beca3b7e3066fe4d 2023-03-30 16:54:20,890 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:20,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:20,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:20,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:20,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:20,893 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:20,894 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/efa49f544bda4991beca3b7e3066fe4d, entries=150, sequenceid=204, filesize=11.8 K 2023-03-30 16:54:20,894 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:20,894 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:20,894 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:20,894 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,894 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,894 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/414b3a60ba9a4f8595599dba6314ddf0 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/414b3a60ba9a4f8595599dba6314ddf0 2023-03-30 16:54:20,898 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:20,898 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:20,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:20,898 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,898 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:20,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,899 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:20,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:20,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:20,899 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:20,899 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:20,899 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:20,900 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:20,900 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:20,900 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:20,900 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:20,900 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:20,900 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:20,900 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:20,900 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:20,900 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,900 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:20,900 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:20,901 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/414b3a60ba9a4f8595599dba6314ddf0, entries=150, sequenceid=204, filesize=11.8 K 2023-03-30 16:54:20,902 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 339a657579649f1a81384cc0c3fc4f47 in 863ms, sequenceid=204, compaction requested=true 2023-03-30 16:54:20,902 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:20,902 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:20,902 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:54:20,902 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:54:20,902 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:20,903 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:20,904 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77996 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:54:20,904 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:54:20,904 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 339a657579649f1a81384cc0c3fc4f47/A is initiating minor compaction (all files) 2023-03-30 16:54:20,904 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 339a657579649f1a81384cc0c3fc4f47/B is initiating minor compaction (all files) 2023-03-30 16:54:20,904 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 339a657579649f1a81384cc0c3fc4f47/A in TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:20,905 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 339a657579649f1a81384cc0c3fc4f47/B in TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:20,905 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/6e7c475539914de38cc5d05ac28927cc, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/a7ae7ba1cbc0406db15c10412dabb987, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/ca52c21756634d7480b8afd80818ce07] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp, totalSize=76.2 K 2023-03-30 16:54:20,905 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/0bc28f09f4ef4283bce5e1479420a847, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/c926aba046134185afa3874591258d2d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/efa49f544bda4991beca3b7e3066fe4d] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp, totalSize=35.6 K 2023-03-30 16:54:20,905 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 6e7c475539914de38cc5d05ac28927cc, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1680195257439 2023-03-30 16:54:20,905 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 0bc28f09f4ef4283bce5e1479420a847, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1680195257439 2023-03-30 16:54:20,905 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting a7ae7ba1cbc0406db15c10412dabb987, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1680195258769 2023-03-30 16:54:20,906 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting c926aba046134185afa3874591258d2d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1680195258769 2023-03-30 16:54:20,905 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,905 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,906 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting ca52c21756634d7480b8afd80818ce07, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1680195259913 2023-03-30 16:54:20,906 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting efa49f544bda4991beca3b7e3066fe4d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1680195259913 2023-03-30 16:54:20,906 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,906 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:20,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:20,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6318b42e[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:20,908 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:20,909 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,910 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,910 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:20,910 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:20,910 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:20,910 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,910 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:20,912 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,912 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:20,912 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,912 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,916 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 339a657579649f1a81384cc0c3fc4f47#B#compaction#129 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:20,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,920 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 339a657579649f1a81384cc0c3fc4f47#A#compaction#130 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:20,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:20,921 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:20,921 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:20,921 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:20,921 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:20,921 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:20,932 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,932 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:20,933 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:20,933 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:20,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:20,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,937 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,940 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:20,941 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:20,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:20,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:20,941 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,941 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,941 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:20,941 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:20,941 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,941 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:20,941 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:20,941 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:20,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:20,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:20,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:20,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:20,942 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:20,942 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:21,042 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:21,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:21,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:21,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:21,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:21,045 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:21,045 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:21,045 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:21,045 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:21,046 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:21,046 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:21,048 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:21,048 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:21,048 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,048 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:21,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:21,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:21,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:21,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:21,049 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:21,049 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:21,049 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:21,049 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:21,050 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:21,050 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:21,050 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:21,050 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-30 16:54:21,050 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:21,050 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:21,050 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:21,050 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:21,050 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:21,050 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:21,067 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330de8c697e987e43b889d5b7e189afc67e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330de8c697e987e43b889d5b7e189afc67e 2023-03-30 16:54:21,072 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=225, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/fba5e5f75b084dc19c0a943b67c6c271 2023-03-30 16:54:21,105 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/2e24fbd2678a46fcbf998fcaefb51892 2023-03-30 16:54:21,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:21,206 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:21,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:21,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195321219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:21,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195321220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:21,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195321323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:21,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195321323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,364 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/41180825a76d4cbcb045f094c73b541f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/41180825a76d4cbcb045f094c73b541f 2023-03-30 16:54:21,370 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 339a657579649f1a81384cc0c3fc4f47/B of 339a657579649f1a81384cc0c3fc4f47 into 41180825a76d4cbcb045f094c73b541f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:21,370 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:21,370 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., storeName=339a657579649f1a81384cc0c3fc4f47/B, priority=13, startTime=1680195260902; duration=0sec 2023-03-30 16:54:21,370 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:21,370 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:54:21,374 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:54:21,374 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 339a657579649f1a81384cc0c3fc4f47/C is initiating minor compaction (all files) 2023-03-30 16:54:21,374 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 339a657579649f1a81384cc0c3fc4f47/C in TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:21,375 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/4ef58e7a82cb480393f818f45c47f71b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/dd2ee5edb19b439bba4b172298117601, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/414b3a60ba9a4f8595599dba6314ddf0] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp, totalSize=35.6 K 2023-03-30 16:54:21,375 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 4ef58e7a82cb480393f818f45c47f71b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1680195257439 2023-03-30 16:54:21,376 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting dd2ee5edb19b439bba4b172298117601, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1680195258769 2023-03-30 16:54:21,376 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 414b3a60ba9a4f8595599dba6314ddf0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1680195259913 2023-03-30 16:54:21,377 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/9cdfc8c547ea446ba4de61bd5b98848f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/9cdfc8c547ea446ba4de61bd5b98848f 2023-03-30 16:54:21,382 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 339a657579649f1a81384cc0c3fc4f47/A of 339a657579649f1a81384cc0c3fc4f47 into 9cdfc8c547ea446ba4de61bd5b98848f(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:21,382 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:21,382 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., storeName=339a657579649f1a81384cc0c3fc4f47/A, priority=13, startTime=1680195260902; duration=0sec 2023-03-30 16:54:21,383 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:21,383 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 339a657579649f1a81384cc0c3fc4f47#C#compaction#134 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:21,526 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:21,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46778 deadline: 1680195321525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:21,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195321525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,551 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/2c69921d7c194fe39046af71538a2c64 2023-03-30 16:54:21,555 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/fba5e5f75b084dc19c0a943b67c6c271 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/fba5e5f75b084dc19c0a943b67c6c271 2023-03-30 16:54:21,559 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/fba5e5f75b084dc19c0a943b67c6c271, entries=150, sequenceid=225, filesize=25.3 K 2023-03-30 16:54:21,560 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/2e24fbd2678a46fcbf998fcaefb51892 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2e24fbd2678a46fcbf998fcaefb51892 2023-03-30 16:54:21,566 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2e24fbd2678a46fcbf998fcaefb51892, entries=150, sequenceid=225, filesize=11.8 K 2023-03-30 16:54:21,567 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/2c69921d7c194fe39046af71538a2c64 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/2c69921d7c194fe39046af71538a2c64 2023-03-30 16:54:21,571 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/2c69921d7c194fe39046af71538a2c64, entries=150, sequenceid=225, filesize=11.8 K 2023-03-30 16:54:21,572 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 339a657579649f1a81384cc0c3fc4f47 in 522ms, sequenceid=225, compaction requested=false 2023-03-30 16:54:21,572 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:21,572 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:21,572 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:21,573 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:21,573 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,573 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:21,573 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:21,574 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,574 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:21,575 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:21,575 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:21,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:21,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50dca6be[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:21,576 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:21,576 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,577 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,578 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:21,578 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:21,578 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:21,578 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,578 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:21,580 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:21,580 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,580 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:21,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:21,580 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:21,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:21,583 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:21,583 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:21,583 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:21,583 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:21,583 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:21,584 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:21,584 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:21,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:21,584 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:21,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:21,585 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:21,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:21,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:21,594 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,596 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:21,597 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:21,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:21,597 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,597 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:21,597 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:21,597 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,597 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,597 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:21,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:21,597 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:21,597 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:21,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:21,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:21,597 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:21,597 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:21,698 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:21,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:21,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:21,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:21,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:21,701 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:21,701 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:21,701 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:21,701 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:21,702 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:21,702 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:21,703 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:21,703 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:21,704 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,704 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:21,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:21,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:21,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:21,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:21,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:21,705 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:21,705 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:21,706 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-30 16:54:21,706 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:21,706 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:21,706 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:21,706 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:21,706 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:21,706 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:21,761 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303304a05ada8ae7e451eb334fd735da6df42 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303304a05ada8ae7e451eb334fd735da6df42 2023-03-30 16:54:21,769 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=239, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/493abf0222f04bf48b972a85fcaa31e5 2023-03-30 16:54:21,785 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/3a886e118a7f42c9921016e33a0f145c 2023-03-30 16:54:21,793 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/7bbeba79d4cb4256be529f871c626130 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/7bbeba79d4cb4256be529f871c626130 2023-03-30 16:54:21,800 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 339a657579649f1a81384cc0c3fc4f47/C of 339a657579649f1a81384cc0c3fc4f47 into 7bbeba79d4cb4256be529f871c626130(size=12.2 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:21,800 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:21,800 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., storeName=339a657579649f1a81384cc0c3fc4f47/C, priority=13, startTime=1680195260903; duration=0sec 2023-03-30 16:54:21,800 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:21,801 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/7c15fc4ddc894fb5b30f627c6a38a50d 2023-03-30 16:54:21,807 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/493abf0222f04bf48b972a85fcaa31e5 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/493abf0222f04bf48b972a85fcaa31e5 2023-03-30 16:54:21,812 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/493abf0222f04bf48b972a85fcaa31e5, entries=150, sequenceid=239, filesize=25.3 K 2023-03-30 16:54:21,814 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/3a886e118a7f42c9921016e33a0f145c as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/3a886e118a7f42c9921016e33a0f145c 2023-03-30 16:54:21,825 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/3a886e118a7f42c9921016e33a0f145c, entries=150, sequenceid=239, filesize=11.8 K 2023-03-30 16:54:21,826 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/7c15fc4ddc894fb5b30f627c6a38a50d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/7c15fc4ddc894fb5b30f627c6a38a50d 2023-03-30 16:54:21,839 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/7c15fc4ddc894fb5b30f627c6a38a50d, entries=150, sequenceid=239, filesize=11.8 K 2023-03-30 16:54:21,843 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 339a657579649f1a81384cc0c3fc4f47 in 137ms, sequenceid=239, compaction requested=true 2023-03-30 16:54:21,843 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:21,843 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:21,846 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,846 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:21,847 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,847 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:21,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:21,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7812963a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:21,849 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:21,849 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,851 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:21,851 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,851 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:21,853 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,853 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:21,853 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:21,853 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:21,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:21,857 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:21,858 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:21,858 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:21,858 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:21,858 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:21,860 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:21,860 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:21,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:21,860 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:21,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:21,861 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:21,862 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,867 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,867 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:21,867 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:21,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:21,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:21,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:21,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:21,867 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:21,867 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:21,867 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,867 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:21,867 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:21,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:21,867 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:21,867 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,868 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,868 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:21,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:21,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:21,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:21,900 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:54:21,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:21,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:21,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:21,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:21,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:21,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:21,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680195261904, max timestamp=1680195261959Num uniques 859; 2023-03-30 16:54:21,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680195261904, max timestamp=1680195261959Num uniques 859; 2023-03-30 16:54:21,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-30 16:54:21,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680195261904, max timestamp=1680195261959Num uniques 859; 2023-03-30 16:54:21,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-30 16:54:21,961 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-30 16:54:21,968 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:21,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:21,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195321965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:21,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:21,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:21,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:21,972 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:21,972 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:21,972 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:21,972 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:21,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:21,973 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:21,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195321970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,973 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:21,975 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:21,975 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:21,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:21,976 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,976 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:21,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:21,977 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:21,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:21,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:21,982 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:21,982 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:21,982 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:21,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:21,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:21,984 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:21,984 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:21,984 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:21,984 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:21,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:21,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:21,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:21,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:21,985 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,985 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:21,986 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,986 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:21,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:21,987 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:21,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76eedc2c[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:21,987 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,989 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,989 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:21,990 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:21,990 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:21,990 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:21,990 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:21,992 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,992 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:21,992 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:21,992 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:22,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:22,000 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:22,000 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:22,000 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:22,000 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:22,000 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:22,002 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,002 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:22,002 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:22,003 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,004 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,011 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,011 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,011 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,011 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,011 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,011 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:22,011 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:22,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:22,012 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,012 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:22,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:22,012 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:22,012 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:22,012 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:22,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:22,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:22,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:22,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195322070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:22,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195322082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,112 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:22,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:22,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:22,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:22,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:22,128 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:22,128 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:22,128 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:22,128 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:22,128 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,129 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,131 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,131 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:22,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:22,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:22,131 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,131 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:22,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:22,133 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:22,133 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:22,133 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:22,133 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:22,133 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:22,133 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:22,133 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:22,133 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:22,133 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:22,133 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:22,133 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:22,133 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:22,133 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,133 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:22,133 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:22,135 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,135 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,135 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,136 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:22,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5071af1b[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:22,137 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:22,137 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,138 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,139 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:22,139 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:22,139 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:22,139 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,139 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:22,140 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:22,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:22,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:22,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:22,143 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:22,143 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:22,143 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:22,143 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:22,143 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:22,147 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,147 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,147 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,150 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,153 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:22,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:22,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:22,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:22,157 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:22,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:22,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:22,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:22,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:22,158 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:22,158 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:22,258 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:22,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:22,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:22,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:22,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:22,261 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:22,261 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:22,261 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:22,261 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:22,264 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,264 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,265 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,265 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:22,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:22,266 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,266 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,266 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:22,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:22,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:22,267 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:22,267 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:22,267 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:22,267 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:22,268 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:22,268 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,268 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:22,268 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:22,269 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,269 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:22,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65ad1afd[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:22,271 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:22,271 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,273 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,273 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:22,273 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,273 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:22,273 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:22,273 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:22,275 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:22,275 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,275 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,275 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:22,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195322275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:22,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:22,282 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:22,282 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:22,282 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:22,282 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:22,282 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:22,284 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,284 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,284 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,284 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:22,285 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195322283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,288 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,288 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,288 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:22,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:22,288 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:22,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:22,288 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:22,288 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:22,289 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:22,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:22,288 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:22,288 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,289 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,289 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,289 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:22,326 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330875bb016250e4e16a35b839be155e463 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330875bb016250e4e16a35b839be155e463 2023-03-30 16:54:22,389 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:22,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:22,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:22,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:22,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:22,392 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:22,392 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:22,392 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:22,392 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:22,393 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,393 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,397 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:22,397 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:22,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:22,398 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,398 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:22,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:22,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:22,399 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:22,399 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:22,399 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:22,399 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:22,399 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:22,399 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:22,399 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:22,400 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:22,400 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:22,400 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:22,401 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,401 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:22,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@962f334[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:22,403 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:22,403 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,404 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:22,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:22,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:22,404 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,404 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:22,406 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:22,406 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:22,406 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,406 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:22,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:22,408 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:22,408 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:22,408 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:22,408 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:22,408 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:22,411 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,411 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:22,411 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:22,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,411 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,413 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,417 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,417 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,417 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,417 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:22,417 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:22,417 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:22,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:22,417 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:22,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:22,417 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:22,417 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:22,417 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,418 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:22,418 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:22,518 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:22,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:22,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:22,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:22,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:22,521 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:22,521 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:22,521 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:22,521 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:22,522 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,522 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,523 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,523 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:22,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:22,523 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:22,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:22,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:22,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:22,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:22,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:22,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:22,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:22,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:22,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:22,526 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:22,526 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:22,526 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:22,526 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:22,526 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:22,526 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:22,526 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,526 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:22,526 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:22,529 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:22,531 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:22,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@96d4188[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:22,531 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,532 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,532 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:22,532 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:22,532 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:22,532 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,532 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:22,534 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:22,534 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:22,534 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:22,535 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:22,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:22,538 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:22,538 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:22,538 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:22,538 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:22,538 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:22,540 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,540 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,540 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,541 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,542 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,545 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,545 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,545 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,545 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,545 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,546 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,545 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:22,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:22,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:22,546 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:22,546 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:22,546 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:22,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:22,546 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:22,546 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:22,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:22,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:22,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195322579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:22,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195322588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,646 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:22,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:22,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:22,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:22,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:22,653 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:22,653 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:22,653 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:22,653 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:22,653 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,653 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,655 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,655 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:22,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:22,655 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,655 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:22,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:22,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:22,663 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:22,663 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:22,663 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:22,666 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:22,666 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:22,666 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:22,666 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:22,668 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:22,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54b93ec9[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:22,670 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:22,670 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,671 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,671 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:22,674 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:22,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:22,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:22,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:22,677 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:22,677 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:22,677 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:22,677 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:22,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:22,683 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:22,683 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:22,683 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:22,683 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:22,684 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:22,686 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,686 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,688 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,691 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,691 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,691 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,691 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:22,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:22,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:22,691 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:22,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,691 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,692 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,692 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,692 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,691 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:22,692 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:22,692 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,692 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:22,692 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:22,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:22,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:22,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:22,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:22,732 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=251, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/7e640522c5bb434dbc9bec071388abea 2023-03-30 16:54:22,792 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:22,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:22,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:22,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:22,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:22,795 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:22,795 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:22,795 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:22,795 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:22,796 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,796 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,797 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:22,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:22,797 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,798 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,798 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:22,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:22,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:22,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:22,799 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:22,799 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:22,799 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:22,799 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:22,800 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:22,800 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,800 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:22,800 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:22,802 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,802 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,803 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,803 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:22,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:22,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59ceac51[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:22,804 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:22,804 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,805 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,806 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:22,806 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:22,806 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:22,806 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,806 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:22,807 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,807 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:22,807 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:22,807 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:22,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:22,809 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:22,809 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:22,809 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:22,809 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:22,809 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:22,811 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,811 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:22,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,811 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,812 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,819 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,823 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,823 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,823 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,823 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,823 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:22,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:22,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:22,823 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:22,824 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:22,823 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,823 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:22,824 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:22,824 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:22,824 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:22,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:22,924 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:22,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:22,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:22,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:22,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:22,928 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:22,928 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:22,928 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:22,928 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:22,929 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,929 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,930 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,930 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:22,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:22,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:22,930 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,930 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:22,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:22,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:22,931 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:22,932 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:22,932 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:22,932 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:22,932 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:22,936 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,936 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,936 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,936 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:22,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@472b1f0f[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:22,938 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:22,938 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,939 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,939 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:22,939 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,940 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:22,940 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:22,940 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:22,941 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,941 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,941 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:22,941 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:22,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:22,943 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:22,943 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:22,943 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:22,943 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:22,943 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:22,946 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,946 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:22,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,946 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:22,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:22,946 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:22,947 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,950 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,950 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:22,950 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:22,950 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:22,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:22,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:22,950 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:22,950 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:22,950 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:22,950 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,950 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:22,950 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:22,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:22,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:22,951 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:22,951 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:23,051 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:23,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:23,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:23,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:23,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:23,054 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:23,054 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:23,054 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:23,054 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:23,055 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,055 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,056 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:23,056 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,057 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,057 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:23,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:23,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:23,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:23,058 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:23,058 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:23,058 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:23,058 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:23,058 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:23,058 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:23,058 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:23,058 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:23,058 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:23,058 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:23,058 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:23,058 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,058 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:23,058 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:23,060 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,060 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:23,061 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,061 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:23,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:23,062 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:23,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24296763[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:23,062 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,064 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,065 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:23,065 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:23,065 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,065 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:23,065 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:23,066 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:23,066 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:23,066 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,066 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:23,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:23,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:23,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:23,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:23,068 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:23,068 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:23,068 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:23,068 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:23,068 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:23,070 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,070 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:23,070 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:23,070 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,072 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:23,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,074 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,074 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,074 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:23,074 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:23,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:23,074 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,074 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,074 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,074 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:23,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:23,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:23,074 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:23,074 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,074 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:23,074 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:23,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:23,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:23,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:23,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195323093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:23,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195323095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,153 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/8c10a7883a2d42fbb1cd448c4db8b3c6 2023-03-30 16:54:23,174 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:23,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:23,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:23,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:23,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:23,177 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:23,177 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:23,177 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:23,178 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:23,178 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,178 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,180 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:23,180 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:23,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:23,180 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,180 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:23,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:23,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:23,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:23,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:23,181 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:23,181 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:23,181 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:23,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:23,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:23,182 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:23,182 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:23,182 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:23,182 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:23,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:23,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:23,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:23,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:23,183 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,183 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:23,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:23,184 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,184 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:23,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:23,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:23,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:23,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:23,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@397b4a59[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:23,185 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:23,185 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,186 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,187 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:23,187 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:23,187 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,187 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:23,187 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:23,188 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,188 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:23,188 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,188 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:23,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:23,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:23,190 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:23,190 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:23,190 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:23,191 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:23,191 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:23,193 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,193 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,193 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:23,194 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:23,196 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,199 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,199 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,199 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:23,199 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:23,199 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,199 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,199 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:23,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:23,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:23,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:23,200 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,200 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,200 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:23,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:23,200 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:23,200 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:23,300 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:23,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:23,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:23,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:23,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:23,303 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:23,303 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:23,303 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:23,304 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:23,304 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,304 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,305 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:23,305 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:23,306 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,306 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:23,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:23,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:23,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:23,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:23,307 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:23,307 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:23,307 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:23,307 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:23,307 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:23,307 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:23,307 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:23,307 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:23,307 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:23,307 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:23,307 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:23,307 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,307 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:23,307 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:23,308 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:23,309 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,310 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,310 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:23,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:23,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@157bd503[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:23,310 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:23,310 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,316 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,316 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:23,316 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:23,316 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:23,316 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,316 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:23,317 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,317 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:23,317 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:23,317 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:23,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:23,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:23,319 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:23,319 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:23,319 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:23,320 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:23,320 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:23,321 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,321 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,322 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:23,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:23,322 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:23,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,323 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:23,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,326 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,326 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,326 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,326 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,326 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,326 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,326 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:23,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:23,326 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:23,326 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:23,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:23,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:23,326 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:23,327 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:23,327 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:23,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:23,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:23,427 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:23,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:23,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:23,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:23,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:23,430 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:23,430 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:23,430 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:23,430 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:23,430 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,430 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,432 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,432 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:23,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:23,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:23,432 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,432 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:23,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:23,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:23,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:23,434 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:23,434 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:23,434 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:23,434 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:23,434 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:23,434 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:23,434 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:23,434 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:23,434 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:23,434 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:23,434 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:23,434 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,434 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:23,434 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:23,435 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,436 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:23,436 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,436 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:23,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:23,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a3763c4[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:23,437 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:23,437 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,438 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,439 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:23,439 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:23,439 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,439 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:23,439 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:23,441 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,441 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:23,441 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:23,442 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:23,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:23,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:23,444 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:23,444 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:23,444 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:23,444 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:23,444 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:23,445 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,445 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:23,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:23,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,445 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:23,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,446 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:23,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:23,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:23,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:23,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,447 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:23,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,449 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:23,449 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:23,449 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:23,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:23,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:23,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:23,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:23,449 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:23,449 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,449 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,449 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,449 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,449 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:23,449 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:23,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:23,449 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,450 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:23,550 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:23,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:23,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:23,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:23,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:23,553 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:23,553 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:23,553 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:23,553 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:23,554 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,554 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,555 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:23,555 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:23,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:23,555 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,555 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:23,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:23,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:23,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:23,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:23,556 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:23,556 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:23,556 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:23,556 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:23,557 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:23,557 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:23,557 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:23,557 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:23,557 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:23,557 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:23,557 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:23,557 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,557 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:23,557 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:23,559 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,559 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:23,559 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,559 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:23,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:23,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65ca1f9e[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:23,560 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:23,561 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,562 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,562 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:23,562 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:23,562 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:23,562 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,562 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:23,564 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:23,564 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:23,564 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:23,564 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:23,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:23,565 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/2fdc1398936c42abb7eaa6c317e901f5 2023-03-30 16:54:23,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:23,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,570 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/7e640522c5bb434dbc9bec071388abea as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/7e640522c5bb434dbc9bec071388abea 2023-03-30 16:54:23,575 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/7e640522c5bb434dbc9bec071388abea, entries=400, sequenceid=251, filesize=59.4 K 2023-03-30 16:54:23,575 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/8c10a7883a2d42fbb1cd448c4db8b3c6 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/8c10a7883a2d42fbb1cd448c4db8b3c6 2023-03-30 16:54:23,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:23,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:23,577 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:23,577 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:23,577 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:23,577 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:23,577 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:23,580 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,580 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,580 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:23,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:23,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:23,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/8c10a7883a2d42fbb1cd448c4db8b3c6, entries=150, sequenceid=251, filesize=11.8 K 2023-03-30 16:54:23,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:23,582 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/2fdc1398936c42abb7eaa6c317e901f5 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/2fdc1398936c42abb7eaa6c317e901f5 2023-03-30 16:54:23,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:23,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:23,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:23,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:23,584 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,585 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,587 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/2fdc1398936c42abb7eaa6c317e901f5, entries=150, sequenceid=251, filesize=11.8 K 2023-03-30 16:54:23,588 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:23,588 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:23,588 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,588 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,588 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,588 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,588 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,588 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:23,588 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:23,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:23,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:23,589 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=147.60 KB/151140 for 339a657579649f1a81384cc0c3fc4f47 in 1689ms, sequenceid=251, compaction requested=true 2023-03-30 16:54:23,589 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:23,589 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:23,589 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:54:23,589 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:23,589 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:54:23,589 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:23,590 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:23,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:23,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:23,590 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:23,590 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:23,592 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48554 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:54:23,592 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 339a657579649f1a81384cc0c3fc4f47/B is initiating minor compaction (all files) 2023-03-30 16:54:23,592 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 339a657579649f1a81384cc0c3fc4f47/B in TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:23,592 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/41180825a76d4cbcb045f094c73b541f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2e24fbd2678a46fcbf998fcaefb51892, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/3a886e118a7f42c9921016e33a0f145c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/8c10a7883a2d42fbb1cd448c4db8b3c6] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp, totalSize=47.4 K 2023-03-30 16:54:23,592 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 138952 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:54:23,592 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 339a657579649f1a81384cc0c3fc4f47/A is initiating minor compaction (all files) 2023-03-30 16:54:23,592 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 339a657579649f1a81384cc0c3fc4f47/A in TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:23,592 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 41180825a76d4cbcb045f094c73b541f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1680195259913 2023-03-30 16:54:23,592 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/9cdfc8c547ea446ba4de61bd5b98848f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/fba5e5f75b084dc19c0a943b67c6c271, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/493abf0222f04bf48b972a85fcaa31e5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/7e640522c5bb434dbc9bec071388abea] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp, totalSize=135.7 K 2023-03-30 16:54:23,593 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 2e24fbd2678a46fcbf998fcaefb51892, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1680195260085 2023-03-30 16:54:23,594 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 9cdfc8c547ea446ba4de61bd5b98848f, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1680195259913 2023-03-30 16:54:23,594 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 3a886e118a7f42c9921016e33a0f145c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680195261215 2023-03-30 16:54:23,595 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting fba5e5f75b084dc19c0a943b67c6c271, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1680195260085 2023-03-30 16:54:23,595 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 8c10a7883a2d42fbb1cd448c4db8b3c6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680195261875 2023-03-30 16:54:23,595 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 493abf0222f04bf48b972a85fcaa31e5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680195261215 2023-03-30 16:54:23,596 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 7e640522c5bb434dbc9bec071388abea, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680195261874 2023-03-30 16:54:23,622 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 339a657579649f1a81384cc0c3fc4f47#B#compaction#142 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:23,625 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 339a657579649f1a81384cc0c3fc4f47#A#compaction#141 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:23,641 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/5153e78eafda4892b64c8cb8ecc5278f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/5153e78eafda4892b64c8cb8ecc5278f 2023-03-30 16:54:23,647 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 339a657579649f1a81384cc0c3fc4f47/B of 339a657579649f1a81384cc0c3fc4f47 into 5153e78eafda4892b64c8cb8ecc5278f(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:23,647 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:23,647 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., storeName=339a657579649f1a81384cc0c3fc4f47/B, priority=12, startTime=1680195263589; duration=0sec 2023-03-30 16:54:23,647 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:23,647 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:54:23,649 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48554 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:54:23,649 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 339a657579649f1a81384cc0c3fc4f47/C is initiating minor compaction (all files) 2023-03-30 16:54:23,649 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 339a657579649f1a81384cc0c3fc4f47/C in TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:23,649 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/7bbeba79d4cb4256be529f871c626130, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/2c69921d7c194fe39046af71538a2c64, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/7c15fc4ddc894fb5b30f627c6a38a50d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/2fdc1398936c42abb7eaa6c317e901f5] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp, totalSize=47.4 K 2023-03-30 16:54:23,650 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 7bbeba79d4cb4256be529f871c626130, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1680195259913 2023-03-30 16:54:23,650 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 2c69921d7c194fe39046af71538a2c64, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1680195260085 2023-03-30 16:54:23,651 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 7c15fc4ddc894fb5b30f627c6a38a50d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1680195261215 2023-03-30 16:54:23,651 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 2fdc1398936c42abb7eaa6c317e901f5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680195261875 2023-03-30 16:54:23,658 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 339a657579649f1a81384cc0c3fc4f47#C#compaction#143 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:23,670 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/0065110449424adcb85902493e0d0386 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/0065110449424adcb85902493e0d0386 2023-03-30 16:54:23,675 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 339a657579649f1a81384cc0c3fc4f47/C of 339a657579649f1a81384cc0c3fc4f47 into 0065110449424adcb85902493e0d0386(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:23,675 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:23,675 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., storeName=339a657579649f1a81384cc0c3fc4f47/C, priority=12, startTime=1680195263589; duration=0sec 2023-03-30 16:54:23,675 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:23,691 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:23,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:23,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:23,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:23,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:23,694 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:23,694 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:23,694 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:23,694 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:23,694 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,694 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,695 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,695 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:23,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:23,696 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:23,696 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:23,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:23,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:23,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:23,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:23,697 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:23,697 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:23,697 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:23,697 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:23,698 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:23,698 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:23,698 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:23,698 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=147.60 KB heapSize=359.20 KB 2023-03-30 16:54:23,698 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:23,698 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:23,698 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:23,698 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:23,698 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:23,698 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:23,712 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033057a3cfad38ef4159b84ec1465d7eace1 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033057a3cfad38ef4159b84ec1465d7eace1 2023-03-30 16:54:24,055 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/dc68baa92e1646f68cd535c53c494983 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/dc68baa92e1646f68cd535c53c494983 2023-03-30 16:54:24,060 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 339a657579649f1a81384cc0c3fc4f47/A of 339a657579649f1a81384cc0c3fc4f47 into dc68baa92e1646f68cd535c53c494983(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:24,060 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:24,060 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., storeName=339a657579649f1a81384cc0c3fc4f47/A, priority=12, startTime=1680195263589; duration=0sec 2023-03-30 16:54:24,060 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:24,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:24,098 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:24,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:24,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195324107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:24,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195324108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,118 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=278, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/8698697015b04b7d8afdba427ec5b5fb 2023-03-30 16:54:24,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:24,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46836 deadline: 1680195324119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,121 DEBUG [Thread-739] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:54:24,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:24,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46776 deadline: 1680195324131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,135 DEBUG [Thread-735] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8189 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:54:24,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:24,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46810 deadline: 1680195324152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,154 DEBUG [Thread-741] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8186 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:54:24,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:24,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:24,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195324211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195324211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:24,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195324415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:24,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195324416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,530 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/54ce7c942918436187a4ace00f0e49e8 2023-03-30 16:54:24,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:24,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195324718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:24,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46760 deadline: 1680195324720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,943 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/b98b21f1bc724e8ba8af849d9caa748b 2023-03-30 16:54:24,947 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/8698697015b04b7d8afdba427ec5b5fb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/8698697015b04b7d8afdba427ec5b5fb 2023-03-30 16:54:24,952 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/8698697015b04b7d8afdba427ec5b5fb, entries=150, sequenceid=278, filesize=25.4 K 2023-03-30 16:54:24,953 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/54ce7c942918436187a4ace00f0e49e8 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/54ce7c942918436187a4ace00f0e49e8 2023-03-30 16:54:24,957 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/54ce7c942918436187a4ace00f0e49e8, entries=150, sequenceid=278, filesize=11.9 K 2023-03-30 16:54:24,958 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/b98b21f1bc724e8ba8af849d9caa748b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/b98b21f1bc724e8ba8af849d9caa748b 2023-03-30 16:54:24,962 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/b98b21f1bc724e8ba8af849d9caa748b, entries=150, sequenceid=278, filesize=11.9 K 2023-03-30 16:54:24,963 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~359.16 KB/367776, currentSize=53.67 KB/54960 for 339a657579649f1a81384cc0c3fc4f47 in 1265ms, sequenceid=278, compaction requested=false 2023-03-30 16:54:24,963 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:24,963 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:24,963 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:24,963 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:24,963 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,963 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:24,963 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:24,966 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:24,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:24,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:24,967 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:24,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:24,967 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:24,967 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:24,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:24,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:24,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:24,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@680b8fc[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:24,968 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:24,969 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:24,970 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:24,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:24,970 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:24,970 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:24,970 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:24,972 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:24,972 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:24,972 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:24,974 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,975 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:24,975 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:24,975 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:24,975 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:24,975 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:24,975 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:24,975 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:24,981 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:24,981 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:24,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:24,981 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:24,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:24,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:24,982 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:24,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:24,983 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,986 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,986 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:24,986 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:24,986 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:24,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:24,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:24,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:24,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:24,986 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:24,986 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:24,986 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:24,986 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:24,986 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:24,986 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:24,986 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:24,986 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:24,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:25,086 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:25,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:25,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:25,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:25,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:25,090 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:25,090 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:25,090 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:25,090 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:25,090 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:25,090 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:25,091 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:25,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:25,091 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:25,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:25,092 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:25,092 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:25,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:25,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:25,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:25,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:25,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:25,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:25,093 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:25,093 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:25,093 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:25,093 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:25,093 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:25,093 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:25,093 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:25,093 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 16:54:25,094 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:25,094 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:25,094 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:25,094 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:25,094 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:25,094 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:25,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:25,226 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:25,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:25,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195325251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:25,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195265236, max timestamp=1680195265254Num uniques 859; 2023-03-30 16:54:25,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195265236, max timestamp=1680195265254Num uniques 859; 2023-03-30 16:54:25,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195265236, max timestamp=1680195265254Num uniques 859; 2023-03-30 16:54:25,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:54:25,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:54:25,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:54:25,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:25,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46760 deadline: 1680195325298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:25,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:25,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195325356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:25,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:25,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46760 deadline: 1680195325402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:25,508 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033080f625c5810d46cdbd5d60c3a7e93145 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033080f625c5810d46cdbd5d60c3a7e93145 2023-03-30 16:54:25,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:25,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195325558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:25,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:25,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46760 deadline: 1680195325606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:25,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:25,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195325863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:25,913 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=290, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/a32394dec0b44336bc357b46801a98a0 2023-03-30 16:54:25,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:25,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46760 deadline: 1680195325909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:25,924 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/2958a0d6158443b19405a70431c43cf0 2023-03-30 16:54:26,337 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/eb4c7d58826a43158da439a6c40ad201 2023-03-30 16:54:26,342 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/a32394dec0b44336bc357b46801a98a0 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/a32394dec0b44336bc357b46801a98a0 2023-03-30 16:54:26,346 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/a32394dec0b44336bc357b46801a98a0, entries=150, sequenceid=290, filesize=25.4 K 2023-03-30 16:54:26,347 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/2958a0d6158443b19405a70431c43cf0 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2958a0d6158443b19405a70431c43cf0 2023-03-30 16:54:26,350 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2958a0d6158443b19405a70431c43cf0, entries=150, sequenceid=290, filesize=11.9 K 2023-03-30 16:54:26,351 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/eb4c7d58826a43158da439a6c40ad201 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/eb4c7d58826a43158da439a6c40ad201 2023-03-30 16:54:26,355 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/eb4c7d58826a43158da439a6c40ad201, entries=150, sequenceid=290, filesize=11.9 K 2023-03-30 16:54:26,356 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 339a657579649f1a81384cc0c3fc4f47 in 1263ms, sequenceid=290, compaction requested=true 2023-03-30 16:54:26,356 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:26,356 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:26,356 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:26,356 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:26,356 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,356 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:26,356 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:26,358 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,358 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:26,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:26,358 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,358 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:26,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:26,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:26,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:26,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:26,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d98a6e2[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:26,359 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:26,359 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,361 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,361 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:26,361 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:26,361 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:26,362 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,362 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:26,363 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,363 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:26,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:26,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:26,363 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:26,363 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:26,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:26,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:26,366 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:26,366 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:26,366 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:26,366 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:26,366 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:26,368 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,368 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:26,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,368 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:26,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:26,368 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:26,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:26,371 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-30 16:54:26,371 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:26,371 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:26,371 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:26,371 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:26,371 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:26,371 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:26,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:26,374 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,377 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:26,377 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:26,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:26,377 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:26,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:26,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:26,377 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,378 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,378 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,378 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,378 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,378 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:26,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:26,378 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:26,378 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:26,378 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:26,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195326386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:26,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46760 deadline: 1680195326416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,478 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:26,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:26,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:26,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:26,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:26,482 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:26,482 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:26,482 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:26,482 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:26,482 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,482 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,485 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,485 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:26,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:26,485 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,485 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:26,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:26,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:26,486 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:26,486 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:26,486 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:26,486 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:26,486 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:26,486 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:26,486 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:26,486 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:26,487 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:26,487 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:26,487 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:26,487 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,487 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:26,487 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:26,488 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,488 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:26,488 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,488 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:26,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:26,490 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:26,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c80d5d6[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:26,490 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,491 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,491 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:26,491 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:26,491 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:26,491 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,491 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:26,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:26,492 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195326490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,492 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:26,492 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:26,492 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:26,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:26,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:26,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:26,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:26,494 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:26,494 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:26,494 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:26,494 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:26,494 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:26,497 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,497 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:26,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,497 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:26,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:26,498 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,499 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,501 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,501 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,501 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,501 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,501 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:26,501 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:26,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:26,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:26,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:26,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:26,501 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,501 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,502 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,501 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:26,502 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:26,502 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:26,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:26,602 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:26,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:26,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:26,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:26,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:26,606 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:26,606 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:26,606 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:26,606 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:26,606 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,606 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,607 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:26,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:26,608 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:26,608 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,608 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:26,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:26,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:26,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:26,609 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:26,609 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:26,609 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:26,609 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:26,609 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:26,609 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:26,609 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:26,609 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:26,609 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:26,610 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:26,610 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:26,610 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,610 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:26,610 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:26,611 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,611 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:26,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:26,612 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,612 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:26,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:26,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:26,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:26,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:26,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49831337[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:26,613 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:26,613 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,616 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,616 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:26,616 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:26,616 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:26,616 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,616 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:26,617 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,617 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:26,617 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:26,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:26,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:26,617 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:26,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:26,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:26,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:26,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:26,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:26,621 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:26,621 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:26,621 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:26,621 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:26,621 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:26,623 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,623 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,623 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:26,623 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,625 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,636 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,636 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,636 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,636 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,636 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,636 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:26,636 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,636 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:26,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:26,637 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:26,637 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:26,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:26,637 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:26,637 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:26,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:26,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:26,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:26,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:26,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195326693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,737 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:26,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:26,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:26,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:26,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:26,740 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:26,740 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:26,740 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:26,740 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:26,741 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,741 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,742 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,742 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:26,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:26,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:26,742 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,742 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:26,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:26,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:26,743 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:26,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:26,744 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:26,744 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:26,744 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:26,744 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:26,744 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:26,744 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:26,744 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:26,744 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:26,744 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:26,744 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:26,744 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:26,744 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,744 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:26,744 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:26,745 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,745 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:26,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:26,746 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,746 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:26,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:26,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:26,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:26,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:26,747 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:26,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ac715a3[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:26,747 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,749 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:26,749 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:26,749 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,749 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:26,749 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:26,751 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,751 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:26,751 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:26,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,751 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:26,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:26,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:26,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:26,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:26,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:26,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:26,754 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:26,754 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:26,754 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:26,754 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:26,754 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:26,756 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,756 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:26,756 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:26,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,756 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:26,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:26,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,757 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:26,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,759 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:26,759 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:26,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:26,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:26,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:26,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:26,760 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,760 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,760 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,760 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,760 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,760 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:26,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:26,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:26,760 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,761 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,761 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:26,761 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:26,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:26,791 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303305eedda653b5b4bd9bb2208799aa56033 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303305eedda653b5b4bd9bb2208799aa56033 2023-03-30 16:54:26,797 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=326, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/8ce54d0bb15841e196e8c3f1143f6cf8 2023-03-30 16:54:26,861 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:26,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:26,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:26,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:26,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:26,864 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:26,864 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:26,864 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:26,864 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:26,865 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,865 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,866 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:26,866 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,867 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,867 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:26,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:26,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:26,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:26,868 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:26,868 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:26,868 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:26,868 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:26,870 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,870 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:26,871 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,871 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:26,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:26,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@426015af[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:26,872 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:26,872 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,873 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,874 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:26,874 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:26,874 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:26,874 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,874 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:26,875 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,875 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:26,875 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:26,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:26,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:26,875 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:26,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:26,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:26,877 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:26,877 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:26,877 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:26,877 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:26,877 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:26,879 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,879 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:26,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:26,879 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:26,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:26,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:26,880 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,881 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,888 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,888 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,888 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,888 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:26,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:26,888 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:26,888 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:26,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:26,888 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:26,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:26,888 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:26,889 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,889 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,889 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:26,889 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:26,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:26,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:26,989 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:26,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:26,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:26,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:26,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:26,992 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:26,992 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:26,992 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:26,992 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:26,993 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,993 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,994 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:26,994 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:26,995 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,995 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:26,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:26,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:26,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:26,996 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:26,996 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:26,996 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:26,996 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:26,996 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:26,996 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:26,996 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:26,996 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:26,996 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:26,996 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:26,996 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:26,996 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,996 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:26,996 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:26,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:26,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195326997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,998 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,998 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:26,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:26,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:27,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:27,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38b44650[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:27,000 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:27,000 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,001 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,001 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:27,001 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,001 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:27,001 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:27,001 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:27,004 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:27,004 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:27,004 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,004 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:27,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:27,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:27,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:27,013 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:27,013 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:27,013 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:27,013 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:27,013 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:27,014 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,014 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,014 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:27,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:27,015 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:27,016 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,021 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:27,021 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,021 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:27,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:27,021 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:27,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:27,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:27,021 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:27,021 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:27,021 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:27,021 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:27,021 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:27,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:27,021 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,022 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,022 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,122 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:27,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:27,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:27,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:27,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:27,125 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:27,125 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:27,125 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:27,125 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:27,125 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,125 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:27,128 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:27,128 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:27,128 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,128 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:27,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:27,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:27,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:27,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:27,130 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:27,130 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:27,130 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:27,130 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:27,130 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:27,130 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:27,130 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:27,130 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:27,130 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:27,130 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:27,130 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:27,130 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,130 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:27,130 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:27,132 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,132 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:27,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:27,132 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,132 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:27,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:27,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:27,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@697ff0da[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:27,133 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:27,134 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,135 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,135 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:27,135 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:27,135 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:27,135 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,135 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:27,136 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,136 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:27,136 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:27,136 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:27,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:27,139 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:27,139 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:27,139 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:27,139 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:27,139 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:27,141 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,141 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,141 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:27,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:27,141 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:27,142 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,145 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,145 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:27,145 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:27,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:27,145 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:27,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:27,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:27,145 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:27,145 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:27,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:27,145 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,145 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:27,145 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:27,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:27,145 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:27,145 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,146 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,209 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/2e42f69594a84ea5b4065c5288adc298 2023-03-30 16:54:27,221 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/3a68063501c94ed18eb70953d382a3a5 2023-03-30 16:54:27,225 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/8ce54d0bb15841e196e8c3f1143f6cf8 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/8ce54d0bb15841e196e8c3f1143f6cf8 2023-03-30 16:54:27,230 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/8ce54d0bb15841e196e8c3f1143f6cf8, entries=200, sequenceid=326, filesize=32.3 K 2023-03-30 16:54:27,231 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/2e42f69594a84ea5b4065c5288adc298 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2e42f69594a84ea5b4065c5288adc298 2023-03-30 16:54:27,235 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2e42f69594a84ea5b4065c5288adc298, entries=150, sequenceid=326, filesize=11.9 K 2023-03-30 16:54:27,240 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/3a68063501c94ed18eb70953d382a3a5 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/3a68063501c94ed18eb70953d382a3a5 2023-03-30 16:54:27,245 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:27,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:27,247 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/3a68063501c94ed18eb70953d382a3a5, entries=150, sequenceid=326, filesize=11.9 K 2023-03-30 16:54:27,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:27,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:27,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:27,251 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:27,251 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:27,252 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:27,252 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:27,254 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,254 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:27,255 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:27,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:27,255 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:27,256 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,256 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:27,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:27,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:27,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:27,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:27,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:27,257 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:27,257 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:27,257 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:27,262 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 339a657579649f1a81384cc0c3fc4f47 in 891ms, sequenceid=326, compaction requested=true 2023-03-30 16:54:27,263 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:27,263 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:27,263 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:27,263 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:54:27,263 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:54:27,263 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:27,264 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:27,264 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:27,264 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:27,264 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:27,264 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:54:27,264 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:27,264 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:27,265 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:27,265 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:27,265 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:27,265 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111620 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:54:27,265 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:27,265 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 339a657579649f1a81384cc0c3fc4f47/A is initiating minor compaction (all files) 2023-03-30 16:54:27,265 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 339a657579649f1a81384cc0c3fc4f47/A in TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:27,265 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/dc68baa92e1646f68cd535c53c494983, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/8698697015b04b7d8afdba427ec5b5fb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/a32394dec0b44336bc357b46801a98a0, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/8ce54d0bb15841e196e8c3f1143f6cf8] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp, totalSize=109.0 K 2023-03-30 16:54:27,265 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49140 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:54:27,265 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 339a657579649f1a81384cc0c3fc4f47/B is initiating minor compaction (all files) 2023-03-30 16:54:27,265 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 339a657579649f1a81384cc0c3fc4f47/B in TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:27,265 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/5153e78eafda4892b64c8cb8ecc5278f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/54ce7c942918436187a4ace00f0e49e8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2958a0d6158443b19405a70431c43cf0, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2e42f69594a84ea5b4065c5288adc298] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp, totalSize=48.0 K 2023-03-30 16:54:27,266 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting dc68baa92e1646f68cd535c53c494983, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680195261875 2023-03-30 16:54:27,266 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 5153e78eafda4892b64c8cb8ecc5278f, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680195261875 2023-03-30 16:54:27,267 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 8698697015b04b7d8afdba427ec5b5fb, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680195261959 2023-03-30 16:54:27,268 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting a32394dec0b44336bc357b46801a98a0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680195264103 2023-03-30 16:54:27,268 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 54ce7c942918436187a4ace00f0e49e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680195261959 2023-03-30 16:54:27,268 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 8ce54d0bb15841e196e8c3f1143f6cf8, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680195265267 2023-03-30 16:54:27,269 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 2958a0d6158443b19405a70431c43cf0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680195264103 2023-03-30 16:54:27,269 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 2e42f69594a84ea5b4065c5288adc298, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680195265267 2023-03-30 16:54:27,278 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 339a657579649f1a81384cc0c3fc4f47#B#compaction#155 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:27,298 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 339a657579649f1a81384cc0c3fc4f47#A#compaction#154 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:27,302 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330078a190b5f744126a494b6b50a881672 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330078a190b5f744126a494b6b50a881672 2023-03-30 16:54:27,308 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/b21775fc8d3045209e9880f0716bf4b1 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/b21775fc8d3045209e9880f0716bf4b1 2023-03-30 16:54:27,313 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 339a657579649f1a81384cc0c3fc4f47/B of 339a657579649f1a81384cc0c3fc4f47 into b21775fc8d3045209e9880f0716bf4b1(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:27,313 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:27,313 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., storeName=339a657579649f1a81384cc0c3fc4f47/B, priority=12, startTime=1680195267263; duration=0sec 2023-03-30 16:54:27,313 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:27,313 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:54:27,315 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49140 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:54:27,315 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 339a657579649f1a81384cc0c3fc4f47/C is initiating minor compaction (all files) 2023-03-30 16:54:27,315 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 339a657579649f1a81384cc0c3fc4f47/C in TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:27,315 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/0065110449424adcb85902493e0d0386, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/b98b21f1bc724e8ba8af849d9caa748b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/eb4c7d58826a43158da439a6c40ad201, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/3a68063501c94ed18eb70953d382a3a5] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp, totalSize=48.0 K 2023-03-30 16:54:27,316 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 0065110449424adcb85902493e0d0386, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680195261875 2023-03-30 16:54:27,316 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=336, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/e9eaecfe2cf04e46acfa6b2ac80104ea 2023-03-30 16:54:27,316 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting b98b21f1bc724e8ba8af849d9caa748b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680195261959 2023-03-30 16:54:27,317 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting eb4c7d58826a43158da439a6c40ad201, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680195264103 2023-03-30 16:54:27,317 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 3a68063501c94ed18eb70953d382a3a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680195265267 2023-03-30 16:54:27,340 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 339a657579649f1a81384cc0c3fc4f47#C#compaction#156 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:27,346 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/92a3dea8deda4123a5c8e6b0fcd1087d 2023-03-30 16:54:27,362 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/d5fdb25b3542446bb30a988047b924af 2023-03-30 16:54:27,367 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/e9eaecfe2cf04e46acfa6b2ac80104ea as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/e9eaecfe2cf04e46acfa6b2ac80104ea 2023-03-30 16:54:27,372 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/e9eaecfe2cf04e46acfa6b2ac80104ea, entries=150, sequenceid=336, filesize=25.4 K 2023-03-30 16:54:27,373 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/92a3dea8deda4123a5c8e6b0fcd1087d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/92a3dea8deda4123a5c8e6b0fcd1087d 2023-03-30 16:54:27,379 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/92a3dea8deda4123a5c8e6b0fcd1087d, entries=150, sequenceid=336, filesize=11.9 K 2023-03-30 16:54:27,380 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/d5fdb25b3542446bb30a988047b924af as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/d5fdb25b3542446bb30a988047b924af 2023-03-30 16:54:27,384 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/d5fdb25b3542446bb30a988047b924af, entries=150, sequenceid=336, filesize=11.9 K 2023-03-30 16:54:27,385 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 339a657579649f1a81384cc0c3fc4f47 in 121ms, sequenceid=336, compaction requested=false 2023-03-30 16:54:27,385 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:27,385 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:27,385 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:27,385 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:27,385 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,385 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:27,385 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:27,387 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,387 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:27,387 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:27,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:27,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:27,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1be78a5b[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:27,390 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:27,390 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,391 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:27,392 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,392 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:27,393 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:27,393 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:27,393 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,393 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:27,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:27,404 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:27,404 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:27,404 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:27,404 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:27,405 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:27,406 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,406 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:27,406 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:27,407 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:27,409 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:27,411 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:27,411 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:27,411 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:27,411 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:27,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:27,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:27,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:27,411 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:27,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:27,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:27,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,412 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:27,412 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:27,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:27,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:27,512 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:27,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:27,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:27,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:27,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:27,520 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:27,520 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:27,520 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:27,520 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:27,520 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,520 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:27,522 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:27,522 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:27,523 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,523 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:27,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:27,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:27,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:27,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:27,524 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:27,524 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:27,525 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-30 16:54:27,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:27,533 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:27,533 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:27,533 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:27,533 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:27,533 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:27,533 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:27,534 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:27,573 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303307d5938f9362641a5a5fda3344bd522cc to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303307d5938f9362641a5a5fda3344bd522cc 2023-03-30 16:54:27,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680195267558, max timestamp=1680195267586Num uniques 859; 2023-03-30 16:54:27,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-30 16:54:27,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680195267558, max timestamp=1680195267586Num uniques 859; 2023-03-30 16:54:27,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-30 16:54:27,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680195267558, max timestamp=1680195267586Num uniques 859; 2023-03-30 16:54:27,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-30 16:54:27,589 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=347, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/e9d826672c1642b18b24c5ac0c2ea560 2023-03-30 16:54:27,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:27,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46760 deadline: 1680195327618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:27,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195327619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:27,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:27,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46760 deadline: 1680195327722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195327723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,751 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/78fa2db6aac54c479da621d6c1a41e42 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/78fa2db6aac54c479da621d6c1a41e42 2023-03-30 16:54:27,754 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/ce2b54929c9842e9b2ce2143d5ce3f56 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/ce2b54929c9842e9b2ce2143d5ce3f56 2023-03-30 16:54:27,756 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 339a657579649f1a81384cc0c3fc4f47/A of 339a657579649f1a81384cc0c3fc4f47 into 78fa2db6aac54c479da621d6c1a41e42(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:27,757 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:27,757 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., storeName=339a657579649f1a81384cc0c3fc4f47/A, priority=12, startTime=1680195267263; duration=0sec 2023-03-30 16:54:27,757 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:27,759 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 339a657579649f1a81384cc0c3fc4f47/C of 339a657579649f1a81384cc0c3fc4f47 into ce2b54929c9842e9b2ce2143d5ce3f56(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:27,759 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:27,759 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., storeName=339a657579649f1a81384cc0c3fc4f47/C, priority=12, startTime=1680195267263; duration=0sec 2023-03-30 16:54:27,759 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:27,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:27,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46760 deadline: 1680195327927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:27,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:27,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195327927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,005 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/af80a6e15f8a400698094af995f7b586 2023-03-30 16:54:28,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:28,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46760 deadline: 1680195328230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,234 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:28,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195328232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,416 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/6cff61eb9177432a86e30d249a659b56 2023-03-30 16:54:28,421 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/e9d826672c1642b18b24c5ac0c2ea560 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/e9d826672c1642b18b24c5ac0c2ea560 2023-03-30 16:54:28,425 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/e9d826672c1642b18b24c5ac0c2ea560, entries=350, sequenceid=347, filesize=52.9 K 2023-03-30 16:54:28,426 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/af80a6e15f8a400698094af995f7b586 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/af80a6e15f8a400698094af995f7b586 2023-03-30 16:54:28,431 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/af80a6e15f8a400698094af995f7b586, entries=150, sequenceid=347, filesize=11.9 K 2023-03-30 16:54:28,431 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/6cff61eb9177432a86e30d249a659b56 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/6cff61eb9177432a86e30d249a659b56 2023-03-30 16:54:28,435 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/6cff61eb9177432a86e30d249a659b56, entries=150, sequenceid=347, filesize=11.9 K 2023-03-30 16:54:28,436 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 339a657579649f1a81384cc0c3fc4f47 in 911ms, sequenceid=347, compaction requested=true 2023-03-30 16:54:28,436 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:28,436 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:28,436 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:28,436 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:28,436 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,438 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:28,438 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:28,440 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:28,440 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:28,440 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:28,440 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:28,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:28,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e829e5[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:28,442 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:28,442 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:28,443 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:28,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:28,443 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:28,443 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:28,443 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:28,443 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,443 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:28,444 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:28,444 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:28,444 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,445 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:28,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:28,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:28,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:28,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:28,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:28,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:28,447 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:28,447 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:28,447 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:28,447 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:28,447 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:28,449 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:28,449 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:28,449 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:28,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:28,450 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:28,451 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,454 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,454 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:28,454 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:28,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:28,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:28,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:28,454 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:28,454 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:28,454 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:28,454 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:28,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:28,454 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:28,454 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,454 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:28,454 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:28,454 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:28,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:28,554 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:28,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:28,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:28,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:28,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:28,558 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:28,558 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:28,558 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:28,558 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:28,558 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:28,558 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:28,560 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:28,560 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:28,560 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,560 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:28,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:28,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:28,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:28,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:28,562 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:28,562 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. started... 2023-03-30 16:54:28,562 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-03-30 16:54:28,563 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:28,563 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:28,563 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:28,563 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:28,563 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:28,563 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:28,584 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033073322d07bc36493bac73f55802e7a001 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033073322d07bc36493bac73f55802e7a001 2023-03-30 16:54:28,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:28,735 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. as already flushing 2023-03-30 16:54:28,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:28,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195328743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:28,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46760 deadline: 1680195328744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:28,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195328847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:28,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46760 deadline: 1680195328847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:28,997 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=384, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/84bc541b909f4c60b023feec7a286c27 2023-03-30 16:54:29,008 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/ffe8aaaeb7874c4990e8dba77247a515 2023-03-30 16:54:29,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:29,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195329048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:29,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:29,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46760 deadline: 1680195329050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:29,078 DEBUG [Thread-744] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a5a81b5 to 127.0.0.1:58435 2023-03-30 16:54:29,078 DEBUG [Thread-744] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:29,080 DEBUG [Thread-746] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45e59be5 to 127.0.0.1:58435 2023-03-30 16:54:29,080 DEBUG [Thread-746] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:29,080 DEBUG [Thread-750] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b416c44 to 127.0.0.1:58435 2023-03-30 16:54:29,080 DEBUG [Thread-750] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:29,085 DEBUG [Thread-748] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2668d41d to 127.0.0.1:58435 2023-03-30 16:54:29,085 DEBUG [Thread-748] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:29,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:29,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46778 deadline: 1680195329352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:29,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:29,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46760 deadline: 1680195329353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:29,419 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/44d70c65a3d141f5a2f9f055a00db082 2023-03-30 16:54:29,424 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/84bc541b909f4c60b023feec7a286c27 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/84bc541b909f4c60b023feec7a286c27 2023-03-30 16:54:29,428 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/84bc541b909f4c60b023feec7a286c27, entries=150, sequenceid=384, filesize=25.4 K 2023-03-30 16:54:29,429 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/ffe8aaaeb7874c4990e8dba77247a515 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/ffe8aaaeb7874c4990e8dba77247a515 2023-03-30 16:54:29,432 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/ffe8aaaeb7874c4990e8dba77247a515, entries=150, sequenceid=384, filesize=11.9 K 2023-03-30 16:54:29,433 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/44d70c65a3d141f5a2f9f055a00db082 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/44d70c65a3d141f5a2f9f055a00db082 2023-03-30 16:54:29,437 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/44d70c65a3d141f5a2f9f055a00db082, entries=150, sequenceid=384, filesize=11.9 K 2023-03-30 16:54:29,437 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 339a657579649f1a81384cc0c3fc4f47 in 875ms, sequenceid=384, compaction requested=true 2023-03-30 16:54:29,438 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:29,438 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:29,439 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:29,439 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:29,440 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:29,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:29,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43f4724e[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:29,441 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:29,441 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:29,442 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:29,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:29,442 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:29,442 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:29,442 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:29,442 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:29,442 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:29,448 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:29,448 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:29,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:29,450 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:29,450 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:29,450 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:29,450 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:29,450 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:29,451 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:29,451 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:29,451 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:29,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:29,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:29,452 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:29,453 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:29,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:29,456 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:29,456 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:29,456 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:29,456 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:29,456 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:29,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:29,456 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:29,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:29,456 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:29,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:29,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:29,456 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:29,456 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:29,456 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:29,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:29,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:29,457 DEBUG [Thread-743] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:29,457 DEBUG [Thread-743] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:29,557 DEBUG [Thread-743] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:29,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:29,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:29,859 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-30 16:54:29,859 DEBUG [Thread-733] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x05609e0c to 127.0.0.1:58435 2023-03-30 16:54:29,859 DEBUG [Thread-737] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7adf5e8e to 127.0.0.1:58435 2023-03-30 16:54:29,859 DEBUG [Thread-733] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:29,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:29,859 DEBUG [Thread-737] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:29,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:29,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:29,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:29,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:29,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:30,274 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033013d406bc3c984b5cbb5320778ceff9c0 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033013d406bc3c984b5cbb5320778ceff9c0 2023-03-30 16:54:30,679 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=396, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/ef06ecd630044f39b7c03db5147f3201 2023-03-30 16:54:31,089 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/27ebd7068e4147c481ef1716d354fdeb 2023-03-30 16:54:31,499 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/ac0677004709402e96eef11a73180110 2023-03-30 16:54:31,503 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/ef06ecd630044f39b7c03db5147f3201 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/ef06ecd630044f39b7c03db5147f3201 2023-03-30 16:54:31,507 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/ef06ecd630044f39b7c03db5147f3201, entries=150, sequenceid=396, filesize=25.4 K 2023-03-30 16:54:31,508 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/27ebd7068e4147c481ef1716d354fdeb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/27ebd7068e4147c481ef1716d354fdeb 2023-03-30 16:54:31,512 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/27ebd7068e4147c481ef1716d354fdeb, entries=150, sequenceid=396, filesize=11.9 K 2023-03-30 16:54:31,512 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/ac0677004709402e96eef11a73180110 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/ac0677004709402e96eef11a73180110 2023-03-30 16:54:31,516 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/ac0677004709402e96eef11a73180110, entries=150, sequenceid=396, filesize=11.9 K 2023-03-30 16:54:31,517 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 339a657579649f1a81384cc0c3fc4f47 in 1659ms, sequenceid=396, compaction requested=true 2023-03-30 16:54:31,517 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:31,517 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:31,517 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:31,517 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 16:54:31,517 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 16:54:31,517 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:31,518 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 159080 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 16:54:31,519 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 339a657579649f1a81384cc0c3fc4f47/A is initiating minor compaction (all files) 2023-03-30 16:54:31,519 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 339a657579649f1a81384cc0c3fc4f47/A in TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:31,519 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/78fa2db6aac54c479da621d6c1a41e42, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/e9eaecfe2cf04e46acfa6b2ac80104ea, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/e9d826672c1642b18b24c5ac0c2ea560, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/84bc541b909f4c60b023feec7a286c27, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/ef06ecd630044f39b7c03db5147f3201] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp, totalSize=155.4 K 2023-03-30 16:54:31,519 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61612 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 16:54:31,519 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 339a657579649f1a81384cc0c3fc4f47/B is initiating minor compaction (all files) 2023-03-30 16:54:31,519 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 339a657579649f1a81384cc0c3fc4f47/B in TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:31,519 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/b21775fc8d3045209e9880f0716bf4b1, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/92a3dea8deda4123a5c8e6b0fcd1087d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/af80a6e15f8a400698094af995f7b586, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/ffe8aaaeb7874c4990e8dba77247a515, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/27ebd7068e4147c481ef1716d354fdeb] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp, totalSize=60.2 K 2023-03-30 16:54:31,519 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 78fa2db6aac54c479da621d6c1a41e42, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680195265267 2023-03-30 16:54:31,520 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting b21775fc8d3045209e9880f0716bf4b1, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680195265267 2023-03-30 16:54:31,520 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting e9eaecfe2cf04e46acfa6b2ac80104ea, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680195266371 2023-03-30 16:54:31,520 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 92a3dea8deda4123a5c8e6b0fcd1087d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680195266371 2023-03-30 16:54:31,520 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting e9d826672c1642b18b24c5ac0c2ea560, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680195267438 2023-03-30 16:54:31,520 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting af80a6e15f8a400698094af995f7b586, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680195267438 2023-03-30 16:54:31,520 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 84bc541b909f4c60b023feec7a286c27, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1680195267610 2023-03-30 16:54:31,521 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting ffe8aaaeb7874c4990e8dba77247a515, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1680195267610 2023-03-30 16:54:31,521 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting ef06ecd630044f39b7c03db5147f3201, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1680195268736 2023-03-30 16:54:31,521 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 27ebd7068e4147c481ef1716d354fdeb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1680195268736 2023-03-30 16:54:31,533 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 339a657579649f1a81384cc0c3fc4f47#B#compaction#169 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:31,535 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 339a657579649f1a81384cc0c3fc4f47#A#compaction#168 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:31,950 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/2b51c61fdffa4b5c8e9b167a41f92619 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2b51c61fdffa4b5c8e9b167a41f92619 2023-03-30 16:54:31,955 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 339a657579649f1a81384cc0c3fc4f47/B of 339a657579649f1a81384cc0c3fc4f47 into 2b51c61fdffa4b5c8e9b167a41f92619(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:31,955 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:31,955 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., storeName=339a657579649f1a81384cc0c3fc4f47/B, priority=11, startTime=1680195271517; duration=0sec 2023-03-30 16:54:31,955 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:31,955 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 16:54:31,956 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61612 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 16:54:31,956 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 339a657579649f1a81384cc0c3fc4f47/C is initiating minor compaction (all files) 2023-03-30 16:54:31,956 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 339a657579649f1a81384cc0c3fc4f47/C in TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:31,957 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/ce2b54929c9842e9b2ce2143d5ce3f56, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/d5fdb25b3542446bb30a988047b924af, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/6cff61eb9177432a86e30d249a659b56, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/44d70c65a3d141f5a2f9f055a00db082, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/ac0677004709402e96eef11a73180110] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp, totalSize=60.2 K 2023-03-30 16:54:31,957 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting ce2b54929c9842e9b2ce2143d5ce3f56, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680195265267 2023-03-30 16:54:31,957 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting d5fdb25b3542446bb30a988047b924af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1680195266371 2023-03-30 16:54:31,957 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 6cff61eb9177432a86e30d249a659b56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680195267438 2023-03-30 16:54:31,958 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 44d70c65a3d141f5a2f9f055a00db082, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1680195267610 2023-03-30 16:54:31,958 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting ac0677004709402e96eef11a73180110, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1680195268736 2023-03-30 16:54:31,967 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 339a657579649f1a81384cc0c3fc4f47#C#compaction#170 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:32,379 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/653b435bb8d84d66b4aaba74234af9b6 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/653b435bb8d84d66b4aaba74234af9b6 2023-03-30 16:54:32,384 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 339a657579649f1a81384cc0c3fc4f47/C of 339a657579649f1a81384cc0c3fc4f47 into 653b435bb8d84d66b4aaba74234af9b6(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:32,384 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:32,384 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., storeName=339a657579649f1a81384cc0c3fc4f47/C, priority=11, startTime=1680195271517; duration=0sec 2023-03-30 16:54:32,385 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:32,760 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/1383ff18a9004764b5ef067722190355 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/1383ff18a9004764b5ef067722190355 2023-03-30 16:54:32,765 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 339a657579649f1a81384cc0c3fc4f47/A of 339a657579649f1a81384cc0c3fc4f47 into 1383ff18a9004764b5ef067722190355(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-30 16:54:32,765 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:32,765 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47., storeName=339a657579649f1a81384cc0c3fc4f47/A, priority=11, startTime=1680195271517; duration=1sec 2023-03-30 16:54:32,765 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:34,167 DEBUG [Thread-739] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52f29016 to 127.0.0.1:58435 2023-03-30 16:54:34,167 DEBUG [Thread-739] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:34,201 DEBUG [Thread-735] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31cba94a to 127.0.0.1:58435 2023-03-30 16:54:34,201 DEBUG [Thread-735] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:34,228 DEBUG [Thread-741] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06f13c14 to 127.0.0.1:58435 2023-03-30 16:54:34,228 DEBUG [Thread-741] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:34,228 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-30 16:54:34,228 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 135 2023-03-30 16:54:34,228 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 27 2023-03-30 16:54:34,228 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 98 2023-03-30 16:54:34,228 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 29 2023-03-30 16:54:34,228 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 30 2023-03-30 16:54:34,228 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-30 16:54:34,228 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(398): read 4891 2023-03-30 16:54:34,228 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(398): read 4764 2023-03-30 16:54:34,228 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-30 16:54:34,229 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(402): scanned 2038 2023-03-30 16:54:34,229 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(403): verified 6114 rows 2023-03-30 16:54:34,229 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(402): scanned 2038 2023-03-30 16:54:34,229 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(403): verified 6114 rows 2023-03-30 16:54:34,229 INFO [Listener at localhost/38075] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-30 16:54:34,229 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58d22e5a to 127.0.0.1:58435 2023-03-30 16:54:34,229 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:34,231 INFO [Listener at localhost/38075] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-30 16:54:34,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-30 16:54:34,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-30 16:54:34,236 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195274236"}]},"ts":"1680195274236"} 2023-03-30 16:54:34,237 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-30 16:54:34,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-30 16:54:34,239 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-30 16:54:34,239 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=339a657579649f1a81384cc0c3fc4f47, UNASSIGN}] 2023-03-30 16:54:34,241 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=339a657579649f1a81384cc0c3fc4f47, UNASSIGN 2023-03-30 16:54:34,241 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=339a657579649f1a81384cc0c3fc4f47, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:34,241 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195274241"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680195274241"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680195274241"}]},"ts":"1680195274241"} 2023-03-30 16:54:34,242 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285}] 2023-03-30 16:54:34,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-30 16:54:34,394 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:34,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 339a657579649f1a81384cc0c3fc4f47, disabling compactions & flushes 2023-03-30 16:54:34,394 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:34,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:34,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. after waiting 0 ms 2023-03-30 16:54:34,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:34,394 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 339a657579649f1a81384cc0c3fc4f47 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-30 16:54:34,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=A 2023-03-30 16:54:34,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:34,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=B 2023-03-30 16:54:34,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:34,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 339a657579649f1a81384cc0c3fc4f47, store=C 2023-03-30 16:54:34,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:34,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-30 16:54:34,806 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033011889e4ff6c14567ad59dd6453f3e85b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033011889e4ff6c14567ad59dd6453f3e85b 2023-03-30 16:54:34,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-30 16:54:35,211 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=405, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/9b2ef993efb549f894403452274ef330 2023-03-30 16:54:35,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-30 16:54:35,607 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-30 16:54:35,621 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/c8289e63d3b64cb5bf93488133d42989 2023-03-30 16:54:36,032 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/c0b904ab423343eebc0c6e483df176ca 2023-03-30 16:54:36,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/A/9b2ef993efb549f894403452274ef330 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/9b2ef993efb549f894403452274ef330 2023-03-30 16:54:36,040 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/9b2ef993efb549f894403452274ef330, entries=150, sequenceid=405, filesize=25.4 K 2023-03-30 16:54:36,041 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/B/c8289e63d3b64cb5bf93488133d42989 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/c8289e63d3b64cb5bf93488133d42989 2023-03-30 16:54:36,044 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/c8289e63d3b64cb5bf93488133d42989, entries=150, sequenceid=405, filesize=11.9 K 2023-03-30 16:54:36,045 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/.tmp/C/c0b904ab423343eebc0c6e483df176ca as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/c0b904ab423343eebc0c6e483df176ca 2023-03-30 16:54:36,048 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/c0b904ab423343eebc0c6e483df176ca, entries=150, sequenceid=405, filesize=11.9 K 2023-03-30 16:54:36,049 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 339a657579649f1a81384cc0c3fc4f47 in 1655ms, sequenceid=405, compaction requested=false 2023-03-30 16:54:36,070 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/ff30f19580ae4a7db821c88ecd4dbf4f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/76934ce5c2b041f3942bf3fdf0f2f1a8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/a2a25068ca5f490ba9619168add2dfea, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/6a8e194d1d6c49e5a62dd0a6f0cfb919, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/b4d928bee68446eeaf32612219e0e5a1, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/832f7695d9b14441987f31e3dc58e248, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/089e79cdb1894cfbafa4dd3fbe323620, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/2f62792d80d24c479e3320d84d9a8b5f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/978a909344ed474b9f6c32db063aad83, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/6e7c475539914de38cc5d05ac28927cc, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/a7ae7ba1cbc0406db15c10412dabb987, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/9cdfc8c547ea446ba4de61bd5b98848f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/ca52c21756634d7480b8afd80818ce07, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/fba5e5f75b084dc19c0a943b67c6c271, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/493abf0222f04bf48b972a85fcaa31e5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/7e640522c5bb434dbc9bec071388abea, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/dc68baa92e1646f68cd535c53c494983, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/8698697015b04b7d8afdba427ec5b5fb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/a32394dec0b44336bc357b46801a98a0, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/8ce54d0bb15841e196e8c3f1143f6cf8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/78fa2db6aac54c479da621d6c1a41e42, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/e9eaecfe2cf04e46acfa6b2ac80104ea, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/e9d826672c1642b18b24c5ac0c2ea560, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/84bc541b909f4c60b023feec7a286c27, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/ef06ecd630044f39b7c03db5147f3201] to archive 2023-03-30 16:54:36,071 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 16:54:36,073 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/ff30f19580ae4a7db821c88ecd4dbf4f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/ff30f19580ae4a7db821c88ecd4dbf4f 2023-03-30 16:54:36,074 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/76934ce5c2b041f3942bf3fdf0f2f1a8 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/76934ce5c2b041f3942bf3fdf0f2f1a8 2023-03-30 16:54:36,076 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/a2a25068ca5f490ba9619168add2dfea to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/a2a25068ca5f490ba9619168add2dfea 2023-03-30 16:54:36,077 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/6a8e194d1d6c49e5a62dd0a6f0cfb919 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/6a8e194d1d6c49e5a62dd0a6f0cfb919 2023-03-30 16:54:36,078 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/b4d928bee68446eeaf32612219e0e5a1 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/b4d928bee68446eeaf32612219e0e5a1 2023-03-30 16:54:36,079 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/832f7695d9b14441987f31e3dc58e248 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/832f7695d9b14441987f31e3dc58e248 2023-03-30 16:54:36,080 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/089e79cdb1894cfbafa4dd3fbe323620 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/089e79cdb1894cfbafa4dd3fbe323620 2023-03-30 16:54:36,081 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/2f62792d80d24c479e3320d84d9a8b5f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/2f62792d80d24c479e3320d84d9a8b5f 2023-03-30 16:54:36,082 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/978a909344ed474b9f6c32db063aad83 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/978a909344ed474b9f6c32db063aad83 2023-03-30 16:54:36,083 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/6e7c475539914de38cc5d05ac28927cc to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/6e7c475539914de38cc5d05ac28927cc 2023-03-30 16:54:36,084 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/a7ae7ba1cbc0406db15c10412dabb987 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/a7ae7ba1cbc0406db15c10412dabb987 2023-03-30 16:54:36,085 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/9cdfc8c547ea446ba4de61bd5b98848f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/9cdfc8c547ea446ba4de61bd5b98848f 2023-03-30 16:54:36,086 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/ca52c21756634d7480b8afd80818ce07 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/ca52c21756634d7480b8afd80818ce07 2023-03-30 16:54:36,087 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/fba5e5f75b084dc19c0a943b67c6c271 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/fba5e5f75b084dc19c0a943b67c6c271 2023-03-30 16:54:36,088 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/493abf0222f04bf48b972a85fcaa31e5 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/493abf0222f04bf48b972a85fcaa31e5 2023-03-30 16:54:36,089 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/7e640522c5bb434dbc9bec071388abea to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/7e640522c5bb434dbc9bec071388abea 2023-03-30 16:54:36,090 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/dc68baa92e1646f68cd535c53c494983 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/dc68baa92e1646f68cd535c53c494983 2023-03-30 16:54:36,091 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/8698697015b04b7d8afdba427ec5b5fb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/8698697015b04b7d8afdba427ec5b5fb 2023-03-30 16:54:36,092 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/a32394dec0b44336bc357b46801a98a0 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/a32394dec0b44336bc357b46801a98a0 2023-03-30 16:54:36,093 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/8ce54d0bb15841e196e8c3f1143f6cf8 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/8ce54d0bb15841e196e8c3f1143f6cf8 2023-03-30 16:54:36,094 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/78fa2db6aac54c479da621d6c1a41e42 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/78fa2db6aac54c479da621d6c1a41e42 2023-03-30 16:54:36,095 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/e9eaecfe2cf04e46acfa6b2ac80104ea to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/e9eaecfe2cf04e46acfa6b2ac80104ea 2023-03-30 16:54:36,096 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/e9d826672c1642b18b24c5ac0c2ea560 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/e9d826672c1642b18b24c5ac0c2ea560 2023-03-30 16:54:36,097 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/84bc541b909f4c60b023feec7a286c27 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/84bc541b909f4c60b023feec7a286c27 2023-03-30 16:54:36,098 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/ef06ecd630044f39b7c03db5147f3201 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/ef06ecd630044f39b7c03db5147f3201 2023-03-30 16:54:36,101 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/71f8120f7bc148c79ec130c304a1d4f5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/fb0e085b67474bc48927096f115f44cc, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/0f16cfa4c0434839aac91d46d715ace9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/784950d216cf4c699d0cefb154f43771, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/9b57e36df4384beda8843ff2f511b351, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/21266e102cb34a12a261ca3d26d192ad, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/ecfe805bf3224321ad9736f8e45cabd2, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/e52e1c84638e404fa151fa1bd04e2756, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/0bc28f09f4ef4283bce5e1479420a847, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2c6eccb49f2f46eea8e51fa4eecea7b7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/c926aba046134185afa3874591258d2d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/41180825a76d4cbcb045f094c73b541f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/efa49f544bda4991beca3b7e3066fe4d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2e24fbd2678a46fcbf998fcaefb51892, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/3a886e118a7f42c9921016e33a0f145c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/5153e78eafda4892b64c8cb8ecc5278f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/8c10a7883a2d42fbb1cd448c4db8b3c6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/54ce7c942918436187a4ace00f0e49e8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2958a0d6158443b19405a70431c43cf0, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/b21775fc8d3045209e9880f0716bf4b1, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2e42f69594a84ea5b4065c5288adc298, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/92a3dea8deda4123a5c8e6b0fcd1087d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/af80a6e15f8a400698094af995f7b586, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/ffe8aaaeb7874c4990e8dba77247a515, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/27ebd7068e4147c481ef1716d354fdeb] to archive 2023-03-30 16:54:36,102 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 16:54:36,104 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/71f8120f7bc148c79ec130c304a1d4f5 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/71f8120f7bc148c79ec130c304a1d4f5 2023-03-30 16:54:36,105 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/fb0e085b67474bc48927096f115f44cc to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/fb0e085b67474bc48927096f115f44cc 2023-03-30 16:54:36,107 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/0f16cfa4c0434839aac91d46d715ace9 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/0f16cfa4c0434839aac91d46d715ace9 2023-03-30 16:54:36,108 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/784950d216cf4c699d0cefb154f43771 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/784950d216cf4c699d0cefb154f43771 2023-03-30 16:54:36,109 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/9b57e36df4384beda8843ff2f511b351 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/9b57e36df4384beda8843ff2f511b351 2023-03-30 16:54:36,110 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/21266e102cb34a12a261ca3d26d192ad to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/21266e102cb34a12a261ca3d26d192ad 2023-03-30 16:54:36,111 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/ecfe805bf3224321ad9736f8e45cabd2 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/ecfe805bf3224321ad9736f8e45cabd2 2023-03-30 16:54:36,111 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/e52e1c84638e404fa151fa1bd04e2756 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/e52e1c84638e404fa151fa1bd04e2756 2023-03-30 16:54:36,113 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/0bc28f09f4ef4283bce5e1479420a847 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/0bc28f09f4ef4283bce5e1479420a847 2023-03-30 16:54:36,114 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2c6eccb49f2f46eea8e51fa4eecea7b7 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2c6eccb49f2f46eea8e51fa4eecea7b7 2023-03-30 16:54:36,115 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/c926aba046134185afa3874591258d2d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/c926aba046134185afa3874591258d2d 2023-03-30 16:54:36,116 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/41180825a76d4cbcb045f094c73b541f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/41180825a76d4cbcb045f094c73b541f 2023-03-30 16:54:36,117 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/efa49f544bda4991beca3b7e3066fe4d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/efa49f544bda4991beca3b7e3066fe4d 2023-03-30 16:54:36,118 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2e24fbd2678a46fcbf998fcaefb51892 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2e24fbd2678a46fcbf998fcaefb51892 2023-03-30 16:54:36,119 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/3a886e118a7f42c9921016e33a0f145c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/3a886e118a7f42c9921016e33a0f145c 2023-03-30 16:54:36,120 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/5153e78eafda4892b64c8cb8ecc5278f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/5153e78eafda4892b64c8cb8ecc5278f 2023-03-30 16:54:36,122 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/8c10a7883a2d42fbb1cd448c4db8b3c6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/8c10a7883a2d42fbb1cd448c4db8b3c6 2023-03-30 16:54:36,123 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/54ce7c942918436187a4ace00f0e49e8 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/54ce7c942918436187a4ace00f0e49e8 2023-03-30 16:54:36,124 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2958a0d6158443b19405a70431c43cf0 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2958a0d6158443b19405a70431c43cf0 2023-03-30 16:54:36,125 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/b21775fc8d3045209e9880f0716bf4b1 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/b21775fc8d3045209e9880f0716bf4b1 2023-03-30 16:54:36,126 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2e42f69594a84ea5b4065c5288adc298 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2e42f69594a84ea5b4065c5288adc298 2023-03-30 16:54:36,127 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/92a3dea8deda4123a5c8e6b0fcd1087d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/92a3dea8deda4123a5c8e6b0fcd1087d 2023-03-30 16:54:36,128 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/af80a6e15f8a400698094af995f7b586 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/af80a6e15f8a400698094af995f7b586 2023-03-30 16:54:36,130 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/ffe8aaaeb7874c4990e8dba77247a515 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/ffe8aaaeb7874c4990e8dba77247a515 2023-03-30 16:54:36,131 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/27ebd7068e4147c481ef1716d354fdeb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/27ebd7068e4147c481ef1716d354fdeb 2023-03-30 16:54:36,134 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/6f346196a2a440daaaf88f8981701228, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/c6b6669a87be44ddac08518de61bb8f4, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/c7bc1edbca2c4137aaf812bef70755ad, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/b7add7088a8448ae924fa779ffea0867, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/b7de9f2a279c4c6cbf1a04b011a9dc4e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/f9b957a3755041e0beae12c143bdcadd, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/bac88a6aeb9244af834887b421ed8bdb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/f31872233b68450db1840a4023c9f2c6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/4ef58e7a82cb480393f818f45c47f71b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/3e8ee5cebdd541f2b33fb0d901b98594, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/dd2ee5edb19b439bba4b172298117601, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/7bbeba79d4cb4256be529f871c626130, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/414b3a60ba9a4f8595599dba6314ddf0, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/2c69921d7c194fe39046af71538a2c64, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/7c15fc4ddc894fb5b30f627c6a38a50d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/0065110449424adcb85902493e0d0386, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/2fdc1398936c42abb7eaa6c317e901f5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/b98b21f1bc724e8ba8af849d9caa748b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/eb4c7d58826a43158da439a6c40ad201, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/ce2b54929c9842e9b2ce2143d5ce3f56, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/3a68063501c94ed18eb70953d382a3a5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/d5fdb25b3542446bb30a988047b924af, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/6cff61eb9177432a86e30d249a659b56, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/44d70c65a3d141f5a2f9f055a00db082, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/ac0677004709402e96eef11a73180110] to archive 2023-03-30 16:54:36,135 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 16:54:36,136 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/6f346196a2a440daaaf88f8981701228 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/6f346196a2a440daaaf88f8981701228 2023-03-30 16:54:36,137 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/c6b6669a87be44ddac08518de61bb8f4 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/c6b6669a87be44ddac08518de61bb8f4 2023-03-30 16:54:36,139 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/c7bc1edbca2c4137aaf812bef70755ad to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/c7bc1edbca2c4137aaf812bef70755ad 2023-03-30 16:54:36,140 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/b7add7088a8448ae924fa779ffea0867 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/b7add7088a8448ae924fa779ffea0867 2023-03-30 16:54:36,141 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/b7de9f2a279c4c6cbf1a04b011a9dc4e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/b7de9f2a279c4c6cbf1a04b011a9dc4e 2023-03-30 16:54:36,142 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/f9b957a3755041e0beae12c143bdcadd to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/f9b957a3755041e0beae12c143bdcadd 2023-03-30 16:54:36,143 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/bac88a6aeb9244af834887b421ed8bdb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/bac88a6aeb9244af834887b421ed8bdb 2023-03-30 16:54:36,144 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/f31872233b68450db1840a4023c9f2c6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/f31872233b68450db1840a4023c9f2c6 2023-03-30 16:54:36,145 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/4ef58e7a82cb480393f818f45c47f71b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/4ef58e7a82cb480393f818f45c47f71b 2023-03-30 16:54:36,146 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/3e8ee5cebdd541f2b33fb0d901b98594 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/3e8ee5cebdd541f2b33fb0d901b98594 2023-03-30 16:54:36,147 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/dd2ee5edb19b439bba4b172298117601 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/dd2ee5edb19b439bba4b172298117601 2023-03-30 16:54:36,149 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/7bbeba79d4cb4256be529f871c626130 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/7bbeba79d4cb4256be529f871c626130 2023-03-30 16:54:36,150 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/414b3a60ba9a4f8595599dba6314ddf0 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/414b3a60ba9a4f8595599dba6314ddf0 2023-03-30 16:54:36,151 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/2c69921d7c194fe39046af71538a2c64 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/2c69921d7c194fe39046af71538a2c64 2023-03-30 16:54:36,152 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/7c15fc4ddc894fb5b30f627c6a38a50d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/7c15fc4ddc894fb5b30f627c6a38a50d 2023-03-30 16:54:36,153 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/0065110449424adcb85902493e0d0386 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/0065110449424adcb85902493e0d0386 2023-03-30 16:54:36,154 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/2fdc1398936c42abb7eaa6c317e901f5 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/2fdc1398936c42abb7eaa6c317e901f5 2023-03-30 16:54:36,155 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/b98b21f1bc724e8ba8af849d9caa748b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/b98b21f1bc724e8ba8af849d9caa748b 2023-03-30 16:54:36,156 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/eb4c7d58826a43158da439a6c40ad201 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/eb4c7d58826a43158da439a6c40ad201 2023-03-30 16:54:36,157 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/ce2b54929c9842e9b2ce2143d5ce3f56 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/ce2b54929c9842e9b2ce2143d5ce3f56 2023-03-30 16:54:36,158 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/3a68063501c94ed18eb70953d382a3a5 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/3a68063501c94ed18eb70953d382a3a5 2023-03-30 16:54:36,159 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/d5fdb25b3542446bb30a988047b924af to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/d5fdb25b3542446bb30a988047b924af 2023-03-30 16:54:36,160 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/6cff61eb9177432a86e30d249a659b56 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/6cff61eb9177432a86e30d249a659b56 2023-03-30 16:54:36,161 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/44d70c65a3d141f5a2f9f055a00db082 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/44d70c65a3d141f5a2f9f055a00db082 2023-03-30 16:54:36,162 DEBUG [StoreCloser-TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/ac0677004709402e96eef11a73180110 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/ac0677004709402e96eef11a73180110 2023-03-30 16:54:36,167 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/recovered.edits/408.seqid, newMaxSeqId=408, maxSeqId=4 2023-03-30 16:54:36,168 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47. 2023-03-30 16:54:36,169 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 339a657579649f1a81384cc0c3fc4f47: 2023-03-30 16:54:36,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:36,171 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=339a657579649f1a81384cc0c3fc4f47, regionState=CLOSED 2023-03-30 16:54:36,171 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195276170"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680195276170"}]},"ts":"1680195276170"} 2023-03-30 16:54:36,173 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-03-30 16:54:36,173 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 339a657579649f1a81384cc0c3fc4f47, server=jenkins-hbase4.apache.org,45193,1680195214285 in 1.9290 sec 2023-03-30 16:54:36,174 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-03-30 16:54:36,174 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=339a657579649f1a81384cc0c3fc4f47, UNASSIGN in 1.9340 sec 2023-03-30 16:54:36,175 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195276174"}]},"ts":"1680195276174"} 2023-03-30 16:54:36,175 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-30 16:54:36,177 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-30 16:54:36,178 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9450 sec 2023-03-30 16:54:36,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-30 16:54:36,342 INFO [Listener at localhost/38075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-03-30 16:54:36,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-30 16:54:36,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:54:36,344 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:54:36,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-30 16:54:36,345 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:54:36,350 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:36,352 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A, FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B, FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C, FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/recovered.edits] 2023-03-30 16:54:36,355 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/1383ff18a9004764b5ef067722190355 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/1383ff18a9004764b5ef067722190355 2023-03-30 16:54:36,356 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/9b2ef993efb549f894403452274ef330 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/A/9b2ef993efb549f894403452274ef330 2023-03-30 16:54:36,359 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2b51c61fdffa4b5c8e9b167a41f92619 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/2b51c61fdffa4b5c8e9b167a41f92619 2023-03-30 16:54:36,360 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/c8289e63d3b64cb5bf93488133d42989 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/B/c8289e63d3b64cb5bf93488133d42989 2023-03-30 16:54:36,362 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/653b435bb8d84d66b4aaba74234af9b6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/653b435bb8d84d66b4aaba74234af9b6 2023-03-30 16:54:36,364 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/c0b904ab423343eebc0c6e483df176ca to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/C/c0b904ab423343eebc0c6e483df176ca 2023-03-30 16:54:36,367 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/recovered.edits/408.seqid to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47/recovered.edits/408.seqid 2023-03-30 16:54:36,368 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/339a657579649f1a81384cc0c3fc4f47 2023-03-30 16:54:36,369 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 16:54:36,369 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-30 16:54:36,370 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-30 16:54:36,373 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033003bab6cdb1744793b8fe71c8e97857c4 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033003bab6cdb1744793b8fe71c8e97857c4 2023-03-30 16:54:36,374 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330078a190b5f744126a494b6b50a881672 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330078a190b5f744126a494b6b50a881672 2023-03-30 16:54:36,376 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033011889e4ff6c14567ad59dd6453f3e85b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033011889e4ff6c14567ad59dd6453f3e85b 2023-03-30 16:54:36,377 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033013d406bc3c984b5cbb5320778ceff9c0 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033013d406bc3c984b5cbb5320778ceff9c0 2023-03-30 16:54:36,378 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303b7713cb9b324f62980f0839e3eca9fa to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303b7713cb9b324f62980f0839e3eca9fa 2023-03-30 16:54:36,379 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033049e374ee56dc46f798bb6d8029ab6ad5 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033049e374ee56dc46f798bb6d8029ab6ad5 2023-03-30 16:54:36,380 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303304a05ada8ae7e451eb334fd735da6df42 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303304a05ada8ae7e451eb334fd735da6df42 2023-03-30 16:54:36,381 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033057a3cfad38ef4159b84ec1465d7eace1 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033057a3cfad38ef4159b84ec1465d7eace1 2023-03-30 16:54:36,382 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303305eedda653b5b4bd9bb2208799aa56033 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303305eedda653b5b4bd9bb2208799aa56033 2023-03-30 16:54:36,383 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303306a4b7bd077584e909a915e2596314330 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303306a4b7bd077584e909a915e2596314330 2023-03-30 16:54:36,384 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033073322d07bc36493bac73f55802e7a001 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033073322d07bc36493bac73f55802e7a001 2023-03-30 16:54:36,385 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330771ab84c6f764b27bb4b346d4723ed90 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330771ab84c6f764b27bb4b346d4723ed90 2023-03-30 16:54:36,387 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303307753d60fa1974a7dbcb7ccb3dcbf69ef to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303307753d60fa1974a7dbcb7ccb3dcbf69ef 2023-03-30 16:54:36,388 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303307d5938f9362641a5a5fda3344bd522cc to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303307d5938f9362641a5a5fda3344bd522cc 2023-03-30 16:54:36,389 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033080f625c5810d46cdbd5d60c3a7e93145 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033080f625c5810d46cdbd5d60c3a7e93145 2023-03-30 16:54:36,391 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330875bb016250e4e16a35b839be155e463 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330875bb016250e4e16a35b839be155e463 2023-03-30 16:54:36,392 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303309715c196ca554a0cab5d1ee798438e18 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303309715c196ca554a0cab5d1ee798438e18 2023-03-30 16:54:36,393 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330a6c7a483853b47a4b9bb65dad9003d45 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330a6c7a483853b47a4b9bb65dad9003d45 2023-03-30 16:54:36,395 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330c3038b15d77a4d92aa4a2a338456cdb9 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330c3038b15d77a4d92aa4a2a338456cdb9 2023-03-30 16:54:36,396 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330da304815d38741de8c916ac40bb1f837 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330da304815d38741de8c916ac40bb1f837 2023-03-30 16:54:36,397 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330de8c697e987e43b889d5b7e189afc67e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330de8c697e987e43b889d5b7e189afc67e 2023-03-30 16:54:36,398 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-30 16:54:36,400 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:54:36,401 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-30 16:54:36,403 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-30 16:54:36,404 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:54:36,404 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-30 16:54:36,404 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680195276404"}]},"ts":"9223372036854775807"} 2023-03-30 16:54:36,405 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-30 16:54:36,406 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 339a657579649f1a81384cc0c3fc4f47, NAME => 'TestAcidGuarantees,,1680195245948.339a657579649f1a81384cc0c3fc4f47.', STARTKEY => '', ENDKEY => ''}] 2023-03-30 16:54:36,406 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-30 16:54:36,406 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680195276406"}]},"ts":"9223372036854775807"} 2023-03-30 16:54:36,407 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-30 16:54:36,409 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:54:36,410 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 67 msec 2023-03-30 16:54:36,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-30 16:54:36,446 INFO [Listener at localhost/38075] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-03-30 16:54:36,459 INFO [Listener at localhost/38075] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=228 (was 234), OpenFileDescriptor=555 (was 579), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=371 (was 349) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=4635 (was 4827) 2023-03-30 16:54:36,470 INFO [Listener at localhost/38075] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=228, OpenFileDescriptor=555, MaxFileDescriptor=60000, SystemLoadAverage=371, ProcessCount=169, AvailableMemoryMB=4635 2023-03-30 16:54:36,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-30 16:54:36,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-30 16:54:36,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-30 16:54:36,474 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-30 16:54:36,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-03-30 16:54:36,475 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-30 16:54:36,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-30 16:54:36,476 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:36,476 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c empty. 2023-03-30 16:54:36,477 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:36,477 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 16:54:36,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-30 16:54:36,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-30 16:54:36,885 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-30 16:54:36,887 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2720c59305e818889603edf8ccad1c2c, NAME => 'TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp 2023-03-30 16:54:36,894 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:54:36,894 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 2720c59305e818889603edf8ccad1c2c, disabling compactions & flushes 2023-03-30 16:54:36,894 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:36,894 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:36,894 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. after waiting 0 ms 2023-03-30 16:54:36,894 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:36,894 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:36,894 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:36,897 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-30 16:54:36,898 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195276898"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680195276898"}]},"ts":"1680195276898"} 2023-03-30 16:54:36,899 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-30 16:54:36,900 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-30 16:54:36,900 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195276900"}]},"ts":"1680195276900"} 2023-03-30 16:54:36,901 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-30 16:54:36,907 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2720c59305e818889603edf8ccad1c2c, ASSIGN}] 2023-03-30 16:54:36,908 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2720c59305e818889603edf8ccad1c2c, ASSIGN 2023-03-30 16:54:36,909 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2720c59305e818889603edf8ccad1c2c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,45193,1680195214285; forceNewPlan=false, retain=false 2023-03-30 16:54:37,060 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=2720c59305e818889603edf8ccad1c2c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:37,060 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195277060"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680195277060"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680195277060"}]},"ts":"1680195277060"} 2023-03-30 16:54:37,062 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285}] 2023-03-30 16:54:37,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-30 16:54:37,217 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:37,218 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2720c59305e818889603edf8ccad1c2c, NAME => 'TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.', STARTKEY => '', ENDKEY => ''} 2023-03-30 16:54:37,218 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:37,218 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:54:37,218 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:37,218 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:37,219 INFO [StoreOpener-2720c59305e818889603edf8ccad1c2c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:37,221 DEBUG [StoreOpener-2720c59305e818889603edf8ccad1c2c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A 2023-03-30 16:54:37,221 DEBUG [StoreOpener-2720c59305e818889603edf8ccad1c2c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A 2023-03-30 16:54:37,222 INFO [StoreOpener-2720c59305e818889603edf8ccad1c2c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:54:37,222 INFO [StoreOpener-2720c59305e818889603edf8ccad1c2c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2720c59305e818889603edf8ccad1c2c columnFamilyName A 2023-03-30 16:54:37,222 INFO [StoreOpener-2720c59305e818889603edf8ccad1c2c-1] regionserver.HStore(310): Store=2720c59305e818889603edf8ccad1c2c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:54:37,222 INFO [StoreOpener-2720c59305e818889603edf8ccad1c2c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:37,223 DEBUG [StoreOpener-2720c59305e818889603edf8ccad1c2c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B 2023-03-30 16:54:37,224 DEBUG [StoreOpener-2720c59305e818889603edf8ccad1c2c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B 2023-03-30 16:54:37,224 INFO [StoreOpener-2720c59305e818889603edf8ccad1c2c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:54:37,224 INFO [StoreOpener-2720c59305e818889603edf8ccad1c2c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2720c59305e818889603edf8ccad1c2c columnFamilyName B 2023-03-30 16:54:37,224 INFO [StoreOpener-2720c59305e818889603edf8ccad1c2c-1] regionserver.HStore(310): Store=2720c59305e818889603edf8ccad1c2c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:54:37,224 INFO [StoreOpener-2720c59305e818889603edf8ccad1c2c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:37,225 DEBUG [StoreOpener-2720c59305e818889603edf8ccad1c2c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C 2023-03-30 16:54:37,225 DEBUG [StoreOpener-2720c59305e818889603edf8ccad1c2c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C 2023-03-30 16:54:37,226 INFO [StoreOpener-2720c59305e818889603edf8ccad1c2c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:54:37,226 INFO [StoreOpener-2720c59305e818889603edf8ccad1c2c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2720c59305e818889603edf8ccad1c2c columnFamilyName C 2023-03-30 16:54:37,226 INFO [StoreOpener-2720c59305e818889603edf8ccad1c2c-1] regionserver.HStore(310): Store=2720c59305e818889603edf8ccad1c2c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:54:37,226 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:37,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:37,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:37,229 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 16:54:37,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:37,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-30 16:54:37,232 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2720c59305e818889603edf8ccad1c2c; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65354205, jitterRate=-0.026146456599235535}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 16:54:37,232 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:37,233 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., pid=30, masterSystemTime=1680195277213 2023-03-30 16:54:37,234 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:37,234 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:37,235 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=2720c59305e818889603edf8ccad1c2c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:37,235 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195277234"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680195277234"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680195277234"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680195277234"}]},"ts":"1680195277234"} 2023-03-30 16:54:37,237 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-03-30 16:54:37,237 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 in 174 msec 2023-03-30 16:54:37,238 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-03-30 16:54:37,238 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2720c59305e818889603edf8ccad1c2c, ASSIGN in 330 msec 2023-03-30 16:54:37,239 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-30 16:54:37,239 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195277239"}]},"ts":"1680195277239"} 2023-03-30 16:54:37,239 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-30 16:54:37,243 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-30 16:54:37,244 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 771 msec 2023-03-30 16:54:37,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-30 16:54:37,578 INFO [Listener at localhost/38075] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-03-30 16:54:37,579 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x23083cf2 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:37,584 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52446f97, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:37,585 DEBUG [Listener at localhost/38075] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:37,586 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34388, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:37,587 DEBUG [Listener at localhost/38075] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-30 16:54:37,588 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47178, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-30 16:54:37,590 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x47772cff to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:37,593 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25fa674f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:37,593 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x53921272 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:37,597 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d7cdc2b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:37,597 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x13544701 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:37,601 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5096c69b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:37,601 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x10b79c2c to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:37,606 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@216aad74, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:37,606 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x5a5a81fa to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:37,609 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76203061, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:37,609 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x3b39d5b8 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:37,612 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c6d5986, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:37,613 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x44f9bd75 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:37,617 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31182447, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:37,617 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x5e5b1c14 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:37,620 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@574000bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:37,621 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x4f29c1c4 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:37,627 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4994df89, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:37,627 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x681ace6c to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:54:37,631 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69bc154a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:54:37,640 DEBUG [hconnection-0x42b142dc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:37,640 DEBUG [hconnection-0x2cc1d1e4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:37,640 DEBUG [hconnection-0x3c015557-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:37,641 DEBUG [hconnection-0x3fd3c283-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:37,644 DEBUG [hconnection-0x2fd84f35-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:37,643 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34414, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:37,643 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34392, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:37,643 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34390, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:37,644 DEBUG [hconnection-0x27700cc9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:37,643 DEBUG [hconnection-0xae759d4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:37,642 DEBUG [hconnection-0x2da7040d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:37,644 DEBUG [hconnection-0x583c0737-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:37,645 DEBUG [hconnection-0x4ddf08d0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:54:37,646 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34430, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:37,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:37,649 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34442, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:37,649 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34466, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:37,655 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34450, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:37,656 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34432, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:37,656 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34482, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:37,656 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34456, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:54:37,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:37,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:37,662 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:37,662 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:37,662 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:37,662 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:37,663 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:37,664 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:37,668 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:37,668 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:37,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:37,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:37,668 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:37,668 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:37,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:37,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:37,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:37,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:37,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:37,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:37,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:37,669 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:37,669 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:37,669 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:37,676 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:37,676 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:37,676 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:37,676 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:37,677 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-30 16:54:37,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:37,699 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:37,699 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:37,699 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:37,699 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:37,699 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:37,699 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:37,699 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:37,713 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/218eaa2da37e4e22a85fc59f18e2b169 2023-03-30 16:54:37,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:37,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195337717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:37,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:37,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34414 deadline: 1680195337717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:37,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:37,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34442 deadline: 1680195337717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:37,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:37,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34482 deadline: 1680195337719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:37,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195277711, max timestamp=1680195277718Num uniques 859; 2023-03-30 16:54:37,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:37,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195277711, max timestamp=1680195277718Num uniques 859; 2023-03-30 16:54:37,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:37,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195277711, max timestamp=1680195277718Num uniques 859; 2023-03-30 16:54:37,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:37,754 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/7d907bd4b7814d2c97c218321e9ec277 2023-03-30 16:54:37,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:37,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34456 deadline: 1680195337774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:37,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:37,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34414 deadline: 1680195337821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:37,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:37,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:37,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34482 deadline: 1680195337822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:37,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34442 deadline: 1680195337822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:37,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:37,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195337822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:37,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:37,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34456 deadline: 1680195337875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34414 deadline: 1680195338023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34482 deadline: 1680195338024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195338024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34442 deadline: 1680195338025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34456 deadline: 1680195338078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,199 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/9862a76ae2074d8b95a2d80e09e8cbf2 2023-03-30 16:54:38,204 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/218eaa2da37e4e22a85fc59f18e2b169 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/218eaa2da37e4e22a85fc59f18e2b169 2023-03-30 16:54:38,209 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/218eaa2da37e4e22a85fc59f18e2b169, entries=100, sequenceid=13, filesize=9.3 K 2023-03-30 16:54:38,210 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/7d907bd4b7814d2c97c218321e9ec277 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/7d907bd4b7814d2c97c218321e9ec277 2023-03-30 16:54:38,214 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/7d907bd4b7814d2c97c218321e9ec277, entries=100, sequenceid=13, filesize=9.3 K 2023-03-30 16:54:38,214 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/9862a76ae2074d8b95a2d80e09e8cbf2 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/9862a76ae2074d8b95a2d80e09e8cbf2 2023-03-30 16:54:38,219 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/9862a76ae2074d8b95a2d80e09e8cbf2, entries=100, sequenceid=13, filesize=9.3 K 2023-03-30 16:54:38,220 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 2720c59305e818889603edf8ccad1c2c in 544ms, sequenceid=13, compaction requested=false 2023-03-30 16:54:38,220 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:38,220 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:38,222 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:38,223 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:38,223 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,223 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:38,223 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:38,225 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,225 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:38,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:38,225 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,225 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:38,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:38,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:38,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:38,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:38,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@551ba3ca[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:38,227 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:38,227 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,228 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,228 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:38,228 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:38,228 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:38,229 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,229 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:38,231 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:38,231 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,231 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:38,231 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:38,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:38,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:38,234 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:38,234 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:38,234 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:38,234 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:38,234 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:38,236 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,236 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:38,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:38,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,236 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:38,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:38,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:38,237 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:38,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:38,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,238 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,245 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,245 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,245 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,245 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:38,245 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:38,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:38,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:38,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:38,245 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:38,245 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:38,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:38,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:38,245 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,245 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:38,245 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,245 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:38,245 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:38,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:38,328 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-30 16:54:38,329 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:38,329 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:38,329 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:38,329 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:38,329 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:38,329 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:38,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34414 deadline: 1680195338334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34482 deadline: 1680195338335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195338335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34442 deadline: 1680195338339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,345 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:38,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:38,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:38,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:38,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:38,349 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:38,349 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:38,349 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:38,349 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:38,350 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,350 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,351 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:38,351 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:38,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:38,352 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,352 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:38,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:38,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:38,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:38,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:38,367 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:38,367 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:38,367 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:38,367 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:38,367 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:38,367 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:38,367 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:38,367 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:38,367 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:38,367 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:38,367 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:38,367 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,367 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:38,367 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:38,369 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,369 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:38,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:38,370 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,370 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:38,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:38,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:38,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:38,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:38,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72894e60[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:38,371 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:38,371 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,372 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,372 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:38,372 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:38,372 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:38,373 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,373 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:38,374 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:38,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:38,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:38,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:38,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:38,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:38,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:38,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:38,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:38,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:38,376 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:38,377 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:38,377 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:38,377 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:38,377 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:38,379 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,379 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:38,379 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:38,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:38,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:38,380 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:38,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:38,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:38,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,381 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:38,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34456 deadline: 1680195338382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,384 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,384 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:38,384 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:38,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:38,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:38,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:38,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:38,384 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,384 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:38,384 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:38,384 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,384 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,384 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,384 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:38,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:38,385 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:38,385 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:38,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34414 deadline: 1680195338438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,440 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34482 deadline: 1680195338438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195338438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34442 deadline: 1680195338441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,485 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:38,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:38,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:38,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:38,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:38,488 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:38,488 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:38,488 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:38,488 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:38,488 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,488 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,489 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,489 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:38,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:38,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:38,490 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,490 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:38,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:38,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:38,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:38,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:38,491 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:38,491 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:38,491 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:38,491 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:38,491 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:38,491 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:38,491 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:38,492 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:38,492 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:38,492 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:38,492 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:38,492 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,492 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:38,492 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:38,493 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,493 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,493 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:38,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:38,493 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:38,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:38,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:38,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:38,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:38,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ab6e6d5[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:38,495 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:38,495 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,497 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,497 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:38,497 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:38,497 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:38,497 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,497 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:38,498 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,498 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:38,499 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:38,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:38,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:38,499 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:38,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:38,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:38,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:38,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:38,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:38,501 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:38,501 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:38,501 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:38,501 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:38,501 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:38,502 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,502 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,502 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:38,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:38,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:38,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:38,503 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:38,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:38,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:38,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,504 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:38,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,506 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:38,506 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:38,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:38,506 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:38,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:38,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:38,506 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:38,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:38,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:38,506 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,507 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:38,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:38,507 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:38,507 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,507 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,507 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,507 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,507 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,607 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:38,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:38,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:38,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:38,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:38,610 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:38,610 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:38,610 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:38,610 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:38,611 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,611 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,613 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:38,613 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:38,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:38,613 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,613 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:38,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:38,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:38,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:38,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:38,614 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:38,615 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:38,615 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:38,615 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:38,615 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:38,615 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:38,615 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:38,615 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:38,615 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:38,615 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:38,615 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:38,615 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,615 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:38,615 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:38,616 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,616 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:38,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:38,617 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,617 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:38,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:38,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:38,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:38,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:38,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a309fcf[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:38,618 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:38,618 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,619 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,619 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:38,619 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:38,619 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:38,619 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,620 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:38,621 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,621 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:38,621 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:38,621 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:38,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:38,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:38,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:38,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:38,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:38,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:38,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:38,623 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:38,623 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:38,623 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:38,623 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:38,623 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:38,631 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,631 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:38,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:38,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,631 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:38,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:38,631 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:38,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:38,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:38,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,632 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:38,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,634 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,634 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:38,635 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:38,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:38,635 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:38,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:38,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:38,635 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:38,635 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:38,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:38,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:38,635 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:38,635 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,635 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:38,635 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,635 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,635 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34414 deadline: 1680195338640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195338641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34482 deadline: 1680195338642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34442 deadline: 1680195338643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,735 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:38,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:38,737 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/46a5965e09a8423583db24b7d60b13f3 2023-03-30 16:54:38,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:38,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:38,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:38,739 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:38,739 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:38,739 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:38,739 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:38,739 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,739 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,740 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,740 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:38,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:38,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:38,741 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,741 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:38,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:38,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:38,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:38,742 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:38,743 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:38,743 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:38,743 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:38,743 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:38,743 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:38,743 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:38,743 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:38,743 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:38,743 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:38,743 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:38,743 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,743 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:38,743 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:38,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,748 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:38,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:38,749 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,749 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:38,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:38,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:38,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:38,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:38,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bd68b0[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:38,751 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:38,751 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,752 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,752 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:38,752 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:38,752 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:38,752 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,752 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:38,753 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,753 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:38,753 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:38,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:38,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:38,753 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:38,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:38,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:38,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:38,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:38,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:38,756 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:38,756 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:38,756 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:38,756 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:38,756 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:38,757 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,757 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:38,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:38,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,757 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:38,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:38,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:38,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:38,758 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:38,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,759 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:38,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,764 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,764 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,764 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:38,764 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:38,764 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:38,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:38,764 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:38,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:38,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:38,765 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,765 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,765 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,765 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:38,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:38,765 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:38,765 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:38,865 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:38,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:38,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:38,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:38,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:38,868 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:38,868 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:38,868 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:38,869 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:38,869 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,869 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,870 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,870 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:38,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:38,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:38,870 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,870 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:38,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:38,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:38,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:38,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:38,872 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:38,872 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:38,872 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:38,872 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:38,872 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:38,872 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:38,872 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:38,872 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:38,872 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:38,872 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:38,872 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:38,872 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,872 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:38,872 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:38,874 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,874 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:38,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:38,875 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,875 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:38,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:38,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:38,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:38,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:38,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a7fccc9[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:38,876 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:38,876 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,880 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,881 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:38,881 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:38,881 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:38,881 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,881 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:38,882 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,882 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:38,882 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:38,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:38,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:38,882 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:38,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:38,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:38,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:38,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:38,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:38,885 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:38,885 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:38,885 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:38,885 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:38,885 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:38,886 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,886 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,886 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:38,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:38,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:38,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:38,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:38,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:38,887 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:38,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,888 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34456 deadline: 1680195338887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:38,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:38,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,890 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,891 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,891 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:38,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:38,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:38,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:38,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:38,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:38,891 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:38,891 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:38,891 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,891 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:38,891 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:38,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:38,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:38,891 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,891 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,891 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:38,891 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34414 deadline: 1680195338943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195338943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34482 deadline: 1680195338945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:38,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34442 deadline: 1680195338948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,991 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:38,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:38,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:38,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:38,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:38,995 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:38,995 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:38,995 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:38,995 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:38,995 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,995 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,997 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:38,997 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:38,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:38,997 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,997 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:38,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:38,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:38,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:38,998 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:38,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:38,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:38,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:38,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:38,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:38,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:38,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:38,999 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:38,999 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:38,999 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:38,999 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:38,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:38,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:38,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:38,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:38,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:39,000 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,000 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:39,001 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,001 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:39,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:39,002 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:39,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67a2151d[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:39,002 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,003 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,003 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:39,003 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:39,003 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,004 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:39,003 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:39,005 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,005 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:39,005 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:39,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:39,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:39,005 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:39,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:39,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:39,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:39,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:39,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:39,007 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:39,007 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:39,007 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:39,007 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:39,007 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:39,010 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,010 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:39,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:39,010 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:39,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:39,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:39,011 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:39,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:39,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,012 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,014 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,014 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:39,014 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:39,014 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,014 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,014 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,014 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:39,015 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:39,015 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:39,015 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:39,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:39,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:39,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:39,015 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:39,015 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:39,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:39,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:39,115 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:39,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:39,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:39,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:39,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:39,119 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:39,119 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:39,119 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:39,119 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:39,120 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,120 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:39,122 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:39,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:39,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:39,122 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,123 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,123 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:39,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:39,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:39,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:39,124 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:39,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:39,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:39,139 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:39,139 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:39,139 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:39,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:39,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:39,140 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:39,140 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:39,140 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:39,140 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:39,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:39,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:39,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:39,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:39,142 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,142 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:39,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:39,143 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,143 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:39,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:39,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:39,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:39,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:39,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69e220c1[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:39,145 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:39,145 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,150 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,150 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,151 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:39,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,151 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:39,151 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:39,151 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:39,151 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/5e2289818caf445a9b1e534b587d4b9b 2023-03-30 16:54:39,153 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:39,153 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:39,153 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:39,156 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:39,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:39,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:39,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:39,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:39,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:39,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:39,163 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:39,163 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:39,163 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:39,163 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:39,163 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:39,166 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,166 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:39,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:39,166 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:39,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:39,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:39,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:39,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,168 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:39,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:39,169 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,174 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:39,174 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:39,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:39,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:39,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:39,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:39,180 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,180 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:39,180 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:39,180 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,180 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,180 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:39,180 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,180 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:39,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:39,181 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:39,181 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:39,200 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/e685969677f64ba5b9ab3171f3b045fc 2023-03-30 16:54:39,204 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/46a5965e09a8423583db24b7d60b13f3 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/46a5965e09a8423583db24b7d60b13f3 2023-03-30 16:54:39,210 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/46a5965e09a8423583db24b7d60b13f3, entries=200, sequenceid=50, filesize=13.9 K 2023-03-30 16:54:39,211 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/5e2289818caf445a9b1e534b587d4b9b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/5e2289818caf445a9b1e534b587d4b9b 2023-03-30 16:54:39,215 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/5e2289818caf445a9b1e534b587d4b9b, entries=150, sequenceid=50, filesize=11.6 K 2023-03-30 16:54:39,217 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/e685969677f64ba5b9ab3171f3b045fc as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/e685969677f64ba5b9ab3171f3b045fc 2023-03-30 16:54:39,233 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/e685969677f64ba5b9ab3171f3b045fc, entries=150, sequenceid=50, filesize=11.6 K 2023-03-30 16:54:39,234 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 2720c59305e818889603edf8ccad1c2c in 906ms, sequenceid=50, compaction requested=false 2023-03-30 16:54:39,234 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:39,281 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:39,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:39,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:39,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:39,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:39,285 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:39,285 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:39,285 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:39,285 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:39,285 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,285 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:39,288 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,288 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:39,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:39,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:39,288 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,288 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:39,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:39,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:39,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:39,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:39,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:39,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:39,289 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:39,289 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:39,289 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:39,290 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:39,290 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:39,290 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:39,290 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:39,290 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:54:39,290 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:39,290 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:39,290 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:39,290 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:39,291 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:39,291 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:39,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:39,446 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:39,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195279449, max timestamp=1680195279484Num uniques 859; 2023-03-30 16:54:39,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195279449, max timestamp=1680195279484Num uniques 859; 2023-03-30 16:54:39,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:39,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:39,485 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195279449, max timestamp=1680195279484Num uniques 859; 2023-03-30 16:54:39,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:39,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:39,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34442 deadline: 1680195339496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:39,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195339496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:39,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34482 deadline: 1680195339498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:39,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34414 deadline: 1680195339499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:39,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34442 deadline: 1680195339600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:39,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195339600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:39,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34482 deadline: 1680195339601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:39,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34414 deadline: 1680195339602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,699 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/8476ffb1dffa44cc8f94ba9a50967f09 2023-03-30 16:54:39,773 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/ffa92e7f50d24f9e891eb9648ac10009 2023-03-30 16:54:39,790 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/77f9182a60df4ebf822dc904985d8f54 2023-03-30 16:54:39,796 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/8476ffb1dffa44cc8f94ba9a50967f09 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/8476ffb1dffa44cc8f94ba9a50967f09 2023-03-30 16:54:39,800 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/8476ffb1dffa44cc8f94ba9a50967f09, entries=100, sequenceid=60, filesize=9.3 K 2023-03-30 16:54:39,800 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/ffa92e7f50d24f9e891eb9648ac10009 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/ffa92e7f50d24f9e891eb9648ac10009 2023-03-30 16:54:39,805 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/ffa92e7f50d24f9e891eb9648ac10009, entries=100, sequenceid=60, filesize=9.3 K 2023-03-30 16:54:39,806 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/77f9182a60df4ebf822dc904985d8f54 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/77f9182a60df4ebf822dc904985d8f54 2023-03-30 16:54:39,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:39,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34442 deadline: 1680195339804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:39,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34482 deadline: 1680195339804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:39,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34414 deadline: 1680195339807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,815 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/77f9182a60df4ebf822dc904985d8f54, entries=100, sequenceid=60, filesize=9.3 K 2023-03-30 16:54:39,816 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2720c59305e818889603edf8ccad1c2c in 526ms, sequenceid=60, compaction requested=true 2023-03-30 16:54:39,816 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:39,816 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:39,816 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:39,817 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:39,817 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,818 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:39,818 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:39,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:39,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195339804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,820 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:39,820 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:39,820 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,820 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:39,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:39,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:39,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@733da5[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:39,822 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:39,822 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,823 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,823 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:39,823 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:39,823 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:39,824 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,824 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:39,825 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:39,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:39,825 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:39,825 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:39,825 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:39,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:39,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:39,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:39,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:39,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:39,828 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:39,829 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:39,829 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:39,829 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:39,829 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:39,832 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,832 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:39,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:39,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,833 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:39,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:39,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:39,834 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:39,835 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:39,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,838 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:39,838 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:39,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:39,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:39,838 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,838 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:39,838 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:39,838 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,838 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,838 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:39,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:39,838 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:39,839 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:39,839 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:39,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:39,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:39,838 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:39,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:39,896 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-03-30 16:54:39,897 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:39,897 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-30 16:54:39,897 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:39,897 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-30 16:54:39,897 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:39,897 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-30 16:54:39,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:39,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34456 deadline: 1680195339914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,939 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:39,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:39,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:39,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:39,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:39,943 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:39,943 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:39,943 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:39,943 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:39,944 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,944 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:39,945 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,945 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:39,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:39,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:39,946 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,946 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:39,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:39,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:39,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:39,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:39,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:39,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:39,947 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:39,947 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:39,947 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:39,948 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:39,948 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:39,948 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:39,948 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:39,948 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:39,948 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:39,948 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:39,948 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:39,948 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,948 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:39,948 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:39,950 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:39,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:39,950 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,950 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,950 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:39,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:39,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:39,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:39,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:39,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@442ea0e6[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:39,962 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:39,962 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,964 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,964 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:39,964 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:39,964 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:39,965 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,965 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:39,966 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,967 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:39,967 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:39,967 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:39,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:39,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:39,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:39,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:39,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:39,970 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:39,970 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:39,970 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:39,970 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:39,970 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:39,972 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,972 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,972 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:39,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:39,972 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:39,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:39,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,974 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:39,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:39,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:39,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:39,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:39,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:39,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:39,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:39,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:39,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:39,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:39,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:39,977 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:39,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,978 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:39,978 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:39,978 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:39,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:39,978 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:40,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:40,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34456 deadline: 1680195340018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,078 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:40,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:40,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:40,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:40,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:40,081 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:40,081 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:40,082 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:40,082 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:40,082 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,082 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,084 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:40,084 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:40,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:40,084 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,084 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:40,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:40,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:40,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:40,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:40,085 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:40,085 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:40,086 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:40,086 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:40,086 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:40,086 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:40,086 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:40,088 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,088 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:40,088 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,088 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:40,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:40,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:40,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33d35f83[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:40,090 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:40,090 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,091 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,091 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,091 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:40,091 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:40,091 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:40,091 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:40,095 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,095 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:40,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,095 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:40,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:40,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:40,096 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:40,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:40,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:40,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:40,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:40,098 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:40,098 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:40,098 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:40,098 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:40,098 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:40,099 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,099 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,100 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:40,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:40,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:40,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,100 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:40,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:40,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:40,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:40,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,101 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,104 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,104 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,104 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,104 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:40,104 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:40,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:40,104 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:40,104 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:40,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:40,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:40,104 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,104 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,104 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,104 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:40,104 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:40,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:40,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:40,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34442 deadline: 1680195340112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:40,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34482 deadline: 1680195340113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:40,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34414 deadline: 1680195340113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:40,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195340120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,205 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:40,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:40,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:40,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:40,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:40,208 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:40,208 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:40,208 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:40,208 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:40,209 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,209 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,210 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:40,210 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:40,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:40,210 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,210 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:40,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:40,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:40,211 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:40,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:40,212 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:40,212 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:40,212 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:40,212 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:40,212 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:40,212 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:40,212 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:40,212 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:40,212 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:40,212 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:40,212 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:40,212 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,212 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:40,212 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:40,215 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,215 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:40,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:40,215 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,215 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:40,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:40,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:40,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:40,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:40,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b4f8510[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:40,216 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:40,217 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,218 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,218 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:40,218 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:40,218 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:40,219 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:40,219 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:40,219 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:40,220 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,220 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:40,220 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:40,220 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:40,220 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:40,221 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:40,221 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:40,221 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:40,222 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,222 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,222 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:40,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:40,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:40,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:40,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:40,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:40,223 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:40,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34456 deadline: 1680195340221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:40,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,225 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:40,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,227 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,227 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:40,228 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:40,228 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,228 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,228 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,228 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,228 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,228 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,228 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:40,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:40,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:40,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:40,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:40,228 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:40,228 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:40,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:40,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:40,310 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/327b00dca9724ea9adc176ea2506243d 2023-03-30 16:54:40,328 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:40,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:40,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:40,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:40,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:40,332 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:40,332 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:40,332 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:40,332 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:40,333 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,333 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,334 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:40,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:40,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:40,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:40,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:40,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:40,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:40,336 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:40,336 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:40,336 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:40,340 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:40,340 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:40,340 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:40,340 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:40,340 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:40,340 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:40,340 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:40,340 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:40,340 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,340 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:40,340 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:40,341 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,341 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,341 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,341 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:40,342 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e7faba5[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:40,342 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:40,342 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:40,342 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,342 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,342 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:40,343 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,343 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:40,343 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:40,343 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:40,343 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,343 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:40,346 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:40,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:40,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:40,346 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:40,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:40,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:40,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:40,349 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:40,349 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:40,349 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:40,349 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:40,349 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:40,350 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,350 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,350 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:40,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:40,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:40,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:40,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:40,351 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:40,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:40,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:40,352 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,354 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,354 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,354 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:40,354 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,355 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:40,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:40,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:40,355 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,355 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:40,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:40,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:40,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:40,355 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,355 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,355 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,355 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:40,355 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:40,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:40,455 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:40,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:40,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:40,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:40,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:40,459 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:40,459 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:40,459 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:40,459 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:40,460 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,460 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,461 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:40,461 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:40,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:40,462 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,462 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:40,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:40,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:40,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:40,463 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:40,463 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:40,463 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:40,463 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:40,476 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:40,476 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:40,476 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:40,476 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:40,476 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:40,476 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:40,476 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:40,476 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:40,476 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,476 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:40,476 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:40,479 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,479 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:40,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:40,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:40,480 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,480 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:40,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:40,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:40,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:40,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@524817c9[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:40,482 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:40,482 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,483 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,483 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:40,483 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:40,483 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:40,483 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,483 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:40,484 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:40,484 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:40,484 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:40,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:40,484 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:40,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:40,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:40,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:40,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:40,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:40,487 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:40,487 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:40,487 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:40,487 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:40,487 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:40,489 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,489 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:40,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,489 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:40,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:40,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:40,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:40,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:40,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,490 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:40,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:40,491 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,496 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,497 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:40,497 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,497 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:40,497 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:40,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:40,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:40,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:40,497 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,497 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:40,497 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:40,497 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,497 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,497 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,497 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:40,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:40,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:40,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:40,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34456 deadline: 1680195340525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,597 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:40,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:40,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:40,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:40,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:40,601 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:40,602 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:40,602 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:40,602 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:40,602 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,602 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,603 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:40,603 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:40,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:40,604 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,604 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:40,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:40,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:40,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:40,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:40,605 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:40,605 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:40,605 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:40,605 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:40,605 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:40,606 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:40,606 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:40,606 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:40,606 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:40,606 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:40,606 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:40,606 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,606 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:40,606 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:40,607 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,607 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:40,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:40,608 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,608 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:40,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:40,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:40,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:40,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:40,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6350a4f6[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:40,609 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:40,609 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,611 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,611 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:40,611 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:40,611 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,611 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:40,611 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:40,613 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,613 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:40,613 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:40,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:40,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:40,613 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:40,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:40,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:40,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:40,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:40,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:40,616 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:40,616 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:40,616 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:40,616 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:40,616 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:40,617 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,617 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,617 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:40,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:40,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:40,618 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:40,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:40,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:40,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:40,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:40,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34414 deadline: 1680195340617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,619 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:40,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:40,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34442 deadline: 1680195340619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:40,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:40,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:40,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:40,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:40,622 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:40,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:40,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:40,622 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:40,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:40,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,622 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:40,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,623 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,623 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,623 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:40,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34482 deadline: 1680195340620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:40,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195340625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,723 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:40,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:40,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:40,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:40,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:40,726 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:40,726 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:40,726 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:40,726 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:40,727 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,727 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,729 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:40,729 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:40,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:40,729 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,729 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:40,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:40,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:40,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:40,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:40,730 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:40,730 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:40,730 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:40,730 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:40,730 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:40,730 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:40,731 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:40,731 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:40,731 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:40,731 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:40,731 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:40,731 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,731 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:40,731 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:40,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:40,733 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/8eaefb7daf8b447783109ff29e603c3c 2023-03-30 16:54:40,733 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:40,733 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,734 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:40,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:40,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:40,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:40,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a56c058[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:40,737 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:40,737 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,739 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,739 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:40,739 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,739 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:40,739 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:40,739 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:40,741 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,741 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:40,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:40,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:40,741 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:40,742 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:40,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:40,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:40,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:40,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:40,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:40,746 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:40,746 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:40,746 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:40,746 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:40,746 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:40,747 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/ad841d7afa9644368cec92b6a941cd93 2023-03-30 16:54:40,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:40,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:40,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:40,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:40,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:40,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:40,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:40,750 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:40,752 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:40,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/327b00dca9724ea9adc176ea2506243d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/327b00dca9724ea9adc176ea2506243d 2023-03-30 16:54:40,757 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:40,757 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:40,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:40,757 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:40,757 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:40,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:40,757 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:40,757 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,757 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,757 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:40,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:40,757 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:40,757 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:40,757 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:40,757 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,760 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/327b00dca9724ea9adc176ea2506243d, entries=150, sequenceid=98, filesize=11.6 K 2023-03-30 16:54:40,761 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/8eaefb7daf8b447783109ff29e603c3c as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/8eaefb7daf8b447783109ff29e603c3c 2023-03-30 16:54:40,765 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/8eaefb7daf8b447783109ff29e603c3c, entries=150, sequenceid=98, filesize=11.6 K 2023-03-30 16:54:40,765 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/ad841d7afa9644368cec92b6a941cd93 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/ad841d7afa9644368cec92b6a941cd93 2023-03-30 16:54:40,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/ad841d7afa9644368cec92b6a941cd93, entries=150, sequenceid=98, filesize=11.6 K 2023-03-30 16:54:40,769 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 2720c59305e818889603edf8ccad1c2c in 873ms, sequenceid=98, compaction requested=true 2023-03-30 16:54:40,769 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:40,769 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:40,769 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:54:40,769 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:54:40,769 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:40,769 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:40,770 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45196 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:54:40,770 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/A is initiating minor compaction (all files) 2023-03-30 16:54:40,771 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/A in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:40,771 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/218eaa2da37e4e22a85fc59f18e2b169, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/46a5965e09a8423583db24b7d60b13f3, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/8476ffb1dffa44cc8f94ba9a50967f09, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/327b00dca9724ea9adc176ea2506243d] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=44.1 K 2023-03-30 16:54:40,771 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 218eaa2da37e4e22a85fc59f18e2b169, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680195277666 2023-03-30 16:54:40,771 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 42856 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:54:40,771 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/B is initiating minor compaction (all files) 2023-03-30 16:54:40,771 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 46a5965e09a8423583db24b7d60b13f3, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680195277769 2023-03-30 16:54:40,771 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/B in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:40,771 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/7d907bd4b7814d2c97c218321e9ec277, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/5e2289818caf445a9b1e534b587d4b9b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/ffa92e7f50d24f9e891eb9648ac10009, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/8eaefb7daf8b447783109ff29e603c3c] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=41.9 K 2023-03-30 16:54:40,772 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 8476ffb1dffa44cc8f94ba9a50967f09, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680195278332 2023-03-30 16:54:40,772 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 7d907bd4b7814d2c97c218321e9ec277, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680195277666 2023-03-30 16:54:40,772 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 327b00dca9724ea9adc176ea2506243d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680195279495 2023-03-30 16:54:40,773 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 5e2289818caf445a9b1e534b587d4b9b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680195277769 2023-03-30 16:54:40,773 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting ffa92e7f50d24f9e891eb9648ac10009, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680195278332 2023-03-30 16:54:40,773 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 8eaefb7daf8b447783109ff29e603c3c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680195279495 2023-03-30 16:54:40,780 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#A#compaction#186 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:40,782 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#B#compaction#187 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:40,797 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/642c29a0c2da44d8b977b0cfba41268f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/642c29a0c2da44d8b977b0cfba41268f 2023-03-30 16:54:40,801 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/B of 2720c59305e818889603edf8ccad1c2c into 642c29a0c2da44d8b977b0cfba41268f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:40,801 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:40,801 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/B, priority=12, startTime=1680195280769; duration=0sec 2023-03-30 16:54:40,801 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:40,802 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:54:40,804 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 42856 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:54:40,804 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/C is initiating minor compaction (all files) 2023-03-30 16:54:40,804 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/C in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:40,804 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/9862a76ae2074d8b95a2d80e09e8cbf2, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/e685969677f64ba5b9ab3171f3b045fc, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/77f9182a60df4ebf822dc904985d8f54, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/ad841d7afa9644368cec92b6a941cd93] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=41.9 K 2023-03-30 16:54:40,805 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 9862a76ae2074d8b95a2d80e09e8cbf2, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680195277666 2023-03-30 16:54:40,806 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting e685969677f64ba5b9ab3171f3b045fc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680195277769 2023-03-30 16:54:40,806 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 77f9182a60df4ebf822dc904985d8f54, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1680195278332 2023-03-30 16:54:40,806 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting ad841d7afa9644368cec92b6a941cd93, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680195279495 2023-03-30 16:54:40,833 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#C#compaction#188 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:40,858 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:40,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:40,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:40,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:40,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:40,861 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:40,861 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:40,861 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:40,861 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:40,862 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,862 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,863 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:40,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:40,863 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,864 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:40,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:40,864 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:40,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:40,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:40,879 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:40,879 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:40,879 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:40,879 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:40,879 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:40,880 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:40,880 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:40,880 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-30 16:54:40,880 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:40,880 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:40,880 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:40,880 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:40,880 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:40,880 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:40,903 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/5d53e13bf81a48e38d96b0363ed68aff 2023-03-30 16:54:41,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:41,033 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:41,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195281059, max timestamp=1680195281087Num uniques 859; 2023-03-30 16:54:41,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195281059, max timestamp=1680195281087Num uniques 859; 2023-03-30 16:54:41,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195281059, max timestamp=1680195281087Num uniques 859; 2023-03-30 16:54:41,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:41,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:41,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:41,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680195281125, max timestamp=1680195281146Num uniques 1018; 2023-03-30 16:54:41,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-30 16:54:41,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680195281125, max timestamp=1680195281146Num uniques 1018; 2023-03-30 16:54:41,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-30 16:54:41,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680195281125, max timestamp=1680195281146Num uniques 1018; 2023-03-30 16:54:41,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-30 16:54:41,192 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/76cd012a853c44118133f59a6fa97f1a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/76cd012a853c44118133f59a6fa97f1a 2023-03-30 16:54:41,198 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/A of 2720c59305e818889603edf8ccad1c2c into 76cd012a853c44118133f59a6fa97f1a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:41,198 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:41,198 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/A, priority=12, startTime=1680195280769; duration=0sec 2023-03-30 16:54:41,198 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:41,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680195281219, max timestamp=1680195281233Num uniques 1027; 2023-03-30 16:54:41,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-30 16:54:41,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680195281219, max timestamp=1680195281233Num uniques 1027; 2023-03-30 16:54:41,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-30 16:54:41,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680195281219, max timestamp=1680195281233Num uniques 1027; 2023-03-30 16:54:41,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-30 16:54:41,263 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/529c276af5b74d6bb6915deb5259aab8 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/529c276af5b74d6bb6915deb5259aab8 2023-03-30 16:54:41,268 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/C of 2720c59305e818889603edf8ccad1c2c into 529c276af5b74d6bb6915deb5259aab8(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:41,268 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:41,268 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/C, priority=12, startTime=1680195280769; duration=0sec 2023-03-30 16:54:41,268 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:41,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680195281277, max timestamp=1680195281290Num uniques 1036; 2023-03-30 16:54:41,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-30 16:54:41,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680195281277, max timestamp=1680195281290Num uniques 1036; 2023-03-30 16:54:41,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680195281277, max timestamp=1680195281290Num uniques 1036; 2023-03-30 16:54:41,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-30 16:54:41,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-30 16:54:41,317 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/fe96aec2f5e64fab939bce6f31023ca1 2023-03-30 16:54:41,327 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/a0ad5b4ae778489aa7de2932b2c92c16 2023-03-30 16:54:41,331 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/5d53e13bf81a48e38d96b0363ed68aff as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/5d53e13bf81a48e38d96b0363ed68aff 2023-03-30 16:54:41,335 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/5d53e13bf81a48e38d96b0363ed68aff, entries=150, sequenceid=108, filesize=11.6 K 2023-03-30 16:54:41,336 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/fe96aec2f5e64fab939bce6f31023ca1 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/fe96aec2f5e64fab939bce6f31023ca1 2023-03-30 16:54:41,340 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/fe96aec2f5e64fab939bce6f31023ca1, entries=150, sequenceid=108, filesize=11.6 K 2023-03-30 16:54:41,341 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/a0ad5b4ae778489aa7de2932b2c92c16 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/a0ad5b4ae778489aa7de2932b2c92c16 2023-03-30 16:54:41,355 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/a0ad5b4ae778489aa7de2932b2c92c16, entries=150, sequenceid=108, filesize=11.6 K 2023-03-30 16:54:41,356 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=120.76 KB/123660 for 2720c59305e818889603edf8ccad1c2c in 476ms, sequenceid=108, compaction requested=false 2023-03-30 16:54:41,356 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:41,356 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:41,356 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:41,357 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:41,357 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,357 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:41,357 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:41,358 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:41,359 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:41,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:41,359 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,359 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:41,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:41,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:41,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d86ab07[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:41,361 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:41,361 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:41,362 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=127.47 KB heapSize=362.53 KB 2023-03-30 16:54:41,362 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:41,362 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:41,362 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:41,362 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:41,362 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:41,362 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:41,363 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,363 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:41,363 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:41,363 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:41,366 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:41,366 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,366 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:41,366 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:41,366 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:41,367 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:41,367 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:41,367 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:41,367 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:41,367 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:41,367 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:41,370 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:41,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:41,370 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,370 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:41,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:41,371 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:41,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:41,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:41,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,372 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,383 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:41,383 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:41,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:41,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:41,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:41,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:41,383 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,383 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:41,384 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:41,384 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,384 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,384 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,384 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:41,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:41,384 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:41,384 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:41,383 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:41,387 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/90c34b1eff734b68a287c4fa65afb21b 2023-03-30 16:54:41,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:41,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34456 deadline: 1680195341405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,459 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/36f56cd004a84956bba20be219503174 2023-03-30 16:54:41,484 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:41,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:41,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:41,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:41,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:41,496 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:41,496 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:41,496 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:41,496 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:41,496 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,497 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:41,498 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:41,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:41,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:41,498 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,499 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:41,499 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:41,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:41,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:41,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:41,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:41,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:41,503 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:41,503 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:41,503 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:41,508 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:41,508 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:41,508 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:41,508 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:41,508 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:41,508 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:41,510 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:41,510 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:41,510 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,510 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:41,510 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:41,512 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:41,512 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/37bc30b9f5d544a9b76ffdbc6404bbae 2023-03-30 16:54:41,512 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34456 deadline: 1680195341511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:41,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:41,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:41,513 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,513 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:41,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:41,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:41,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:41,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f621d21[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:41,514 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:41,514 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,517 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,517 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,517 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:41,517 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:41,517 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:41,517 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:41,517 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/90c34b1eff734b68a287c4fa65afb21b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/90c34b1eff734b68a287c4fa65afb21b 2023-03-30 16:54:41,518 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,518 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:41,519 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:41,519 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:41,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:41,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:41,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:41,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:41,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:41,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:41,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:41,521 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:41,521 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:41,521 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:41,521 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:41,521 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:41,522 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/90c34b1eff734b68a287c4fa65afb21b, entries=150, sequenceid=197, filesize=11.8 K 2023-03-30 16:54:41,523 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/36f56cd004a84956bba20be219503174 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/36f56cd004a84956bba20be219503174 2023-03-30 16:54:41,525 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,526 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:41,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,526 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:41,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:41,526 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:41,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:41,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:41,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:41,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,528 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:41,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,530 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/36f56cd004a84956bba20be219503174, entries=150, sequenceid=197, filesize=11.8 K 2023-03-30 16:54:41,530 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,530 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:41,530 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:41,530 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,530 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:41,530 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:41,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:41,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:41,530 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:41,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:41,530 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,531 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,530 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:41,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:41,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:41,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:41,531 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:41,531 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:41,531 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/37bc30b9f5d544a9b76ffdbc6404bbae as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/37bc30b9f5d544a9b76ffdbc6404bbae 2023-03-30 16:54:41,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/37bc30b9f5d544a9b76ffdbc6404bbae, entries=150, sequenceid=197, filesize=11.8 K 2023-03-30 16:54:41,536 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~362.48 KB/371184, currentSize=53.67 KB/54960 for 2720c59305e818889603edf8ccad1c2c in 174ms, sequenceid=197, compaction requested=true 2023-03-30 16:54:41,538 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:41,538 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:41,538 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:54:41,538 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:54:41,538 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:41,538 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:41,539 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:54:41,539 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:54:41,539 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/A is initiating minor compaction (all files) 2023-03-30 16:54:41,539 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/B is initiating minor compaction (all files) 2023-03-30 16:54:41,539 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/A in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:41,539 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/B in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:41,540 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/76cd012a853c44118133f59a6fa97f1a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/5d53e13bf81a48e38d96b0363ed68aff, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/90c34b1eff734b68a287c4fa65afb21b] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=35.1 K 2023-03-30 16:54:41,540 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/642c29a0c2da44d8b977b0cfba41268f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/fe96aec2f5e64fab939bce6f31023ca1, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/36f56cd004a84956bba20be219503174] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=35.1 K 2023-03-30 16:54:41,540 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 76cd012a853c44118133f59a6fa97f1a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680195279495 2023-03-30 16:54:41,540 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 642c29a0c2da44d8b977b0cfba41268f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680195279495 2023-03-30 16:54:41,540 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 5d53e13bf81a48e38d96b0363ed68aff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680195279907 2023-03-30 16:54:41,541 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting fe96aec2f5e64fab939bce6f31023ca1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680195279907 2023-03-30 16:54:41,541 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 90c34b1eff734b68a287c4fa65afb21b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680195281312 2023-03-30 16:54:41,541 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 36f56cd004a84956bba20be219503174, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680195281312 2023-03-30 16:54:41,551 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#B#compaction#195 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:41,557 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#A#compaction#196 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:41,596 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/62c78c882a27414f9e000303debb4a3e as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/62c78c882a27414f9e000303debb4a3e 2023-03-30 16:54:41,610 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/B of 2720c59305e818889603edf8ccad1c2c into 62c78c882a27414f9e000303debb4a3e(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:41,610 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:41,610 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/B, priority=13, startTime=1680195281538; duration=0sec 2023-03-30 16:54:41,611 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:41,611 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:54:41,612 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:54:41,612 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/C is initiating minor compaction (all files) 2023-03-30 16:54:41,612 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/C in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:41,612 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/529c276af5b74d6bb6915deb5259aab8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/a0ad5b4ae778489aa7de2932b2c92c16, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/37bc30b9f5d544a9b76ffdbc6404bbae] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=35.1 K 2023-03-30 16:54:41,612 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 529c276af5b74d6bb6915deb5259aab8, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680195279495 2023-03-30 16:54:41,613 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting a0ad5b4ae778489aa7de2932b2c92c16, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680195279907 2023-03-30 16:54:41,613 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 37bc30b9f5d544a9b76ffdbc6404bbae, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680195281312 2023-03-30 16:54:41,621 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#C#compaction#197 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:41,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:41,626 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-30 16:54:41,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:41,626 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:41,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:41,626 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:41,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:41,626 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:41,631 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:41,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:41,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:41,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:41,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:41,648 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:41,648 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:41,648 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:41,648 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:41,649 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,649 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:41,650 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:41,650 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:41,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:41,651 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,651 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:41,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:41,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:41,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:41,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:41,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:41,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195341651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195281647, max timestamp=1680195281656Num uniques 859; 2023-03-30 16:54:41,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:54:41,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195281647, max timestamp=1680195281656Num uniques 859; 2023-03-30 16:54:41,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:54:41,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195281647, max timestamp=1680195281656Num uniques 859; 2023-03-30 16:54:41,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:54:41,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:41,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34414 deadline: 1680195341691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:41,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34482 deadline: 1680195341691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:41,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:41,700 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:41,700 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:41,700 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:41,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:41,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34442 deadline: 1680195341697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,703 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-30 16:54:41,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:41,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34456 deadline: 1680195341715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,732 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:41,732 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:41,732 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:41,732 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:41,734 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:41,734 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:41,734 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:41,734 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:41,734 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,734 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:41,734 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:41,736 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,736 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,736 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:41,739 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:41,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:41,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:41,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:41,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:41,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:41,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9f3632f[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:41,742 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:41,742 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,744 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,744 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,744 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:41,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,744 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:41,744 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:41,744 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:41,746 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:41,746 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,746 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:41,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:41,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:41,746 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:41,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:41,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:41,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:41,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:41,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:41,749 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:41,749 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:41,749 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:41,749 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:41,749 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:41,751 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,751 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:41,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:41,751 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:41,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:41,751 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:41,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:41,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:41,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,753 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:41,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,755 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,755 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:41,755 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:41,755 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,755 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:41,755 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,755 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:41,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:41,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:41,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:41,755 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:41,755 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,756 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,756 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:41,756 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:41,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:41,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:41,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:41,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:41,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:41,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195341757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:41,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34482 deadline: 1680195341798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:41,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34414 deadline: 1680195341798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:41,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34442 deadline: 1680195341803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,856 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:41,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:41,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:41,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:41,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:41,868 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:41,868 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:41,868 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:41,868 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:41,869 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,869 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:41,871 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:41,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:41,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:41,872 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:41,872 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,872 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:41,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:41,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:41,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:41,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:41,892 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:41,892 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:41,892 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:41,892 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:41,892 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:41,892 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:41,892 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:41,892 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:41,893 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:41,893 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:41,893 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:41,893 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:41,893 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,893 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:41,893 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:41,894 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,895 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,895 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:41,895 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:41,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:41,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:41,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755b3649[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:41,897 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:41,897 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,898 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,898 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:41,898 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:41,898 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:41,898 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,898 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:41,899 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:41,899 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:41,899 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:41,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:41,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:41,903 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:41,903 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:41,903 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:41,903 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:41,903 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:41,907 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,907 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:41,907 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:41,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:41,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:41,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:41,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:41,908 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:41,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,909 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,912 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,912 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:41,912 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:41,912 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,912 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,912 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:41,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:41,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:41,912 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:41,912 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:41,912 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:41,912 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:41,913 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:41,912 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:41,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:41,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:41,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:41,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:41,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:41,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195341964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:41,993 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/1c29b57a95d945f9a7e3a25f2be4cb49 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/1c29b57a95d945f9a7e3a25f2be4cb49 2023-03-30 16:54:41,998 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/A of 2720c59305e818889603edf8ccad1c2c into 1c29b57a95d945f9a7e3a25f2be4cb49(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:41,998 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:41,998 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/A, priority=13, startTime=1680195281538; duration=0sec 2023-03-30 16:54:41,999 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:42,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:42,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:42,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34414 deadline: 1680195342002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34482 deadline: 1680195342003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:42,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34442 deadline: 1680195342008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,013 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:42,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:42,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:42,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:42,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:42,016 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:42,016 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:42,016 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:42,016 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:42,016 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,016 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,020 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:42,020 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:42,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:42,020 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,020 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:42,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:42,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:42,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:42,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:42,022 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:42,022 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:42,022 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:42,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:42,024 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:42,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34456 deadline: 1680195342021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,024 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:42,024 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:42,024 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:42,024 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:42,024 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:42,024 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:42,025 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:42,025 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,025 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:42,025 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:42,026 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:42,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:42,027 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:42,027 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,027 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:42,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:42,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:42,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:42,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d4d6576[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:42,032 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:42,032 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,034 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,034 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:42,034 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:42,034 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:42,034 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,034 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:42,035 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/3a85f5f248e54cf08b709d9b1647f542 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/3a85f5f248e54cf08b709d9b1647f542 2023-03-30 16:54:42,038 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,038 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:42,038 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:42,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:42,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:42,038 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:42,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:42,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:42,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:42,042 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/C of 2720c59305e818889603edf8ccad1c2c into 3a85f5f248e54cf08b709d9b1647f542(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:42,042 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:42,042 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/C, priority=13, startTime=1680195281538; duration=0sec 2023-03-30 16:54:42,042 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:42,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:42,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:42,043 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:42,043 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:42,043 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:42,043 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:42,043 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:42,045 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:42,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:42,045 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,045 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:42,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:42,046 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:42,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:42,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:42,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,050 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:42,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,054 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:42,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:42,054 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:42,054 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:42,054 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:42,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:42,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:42,054 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:42,054 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,054 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,054 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,054 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:42,054 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,054 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,054 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:42,128 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/b8f1d38a233646e98a4e50d3569b9194 2023-03-30 16:54:42,154 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:42,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:42,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:42,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:42,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:42,158 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:42,158 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:42,158 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:42,158 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:42,158 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,158 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,159 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,159 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:42,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:42,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:42,159 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,160 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:42,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:42,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:42,161 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:42,161 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:42,169 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:42,169 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:42,169 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:42,169 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:42,169 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:42,169 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:42,170 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:42,170 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:42,170 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:42,170 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:42,170 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:42,170 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,170 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:42,170 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:42,173 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,173 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:42,173 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,173 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:42,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:42,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:42,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4239fceb[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:42,174 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:42,174 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,175 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,175 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:42,176 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:42,176 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:42,176 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,176 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:42,177 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,177 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:42,177 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:42,177 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:42,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:42,179 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:42,179 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:42,179 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:42,179 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:42,179 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:42,181 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,181 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:42,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:42,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,181 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:42,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:42,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:42,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:42,181 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:42,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,183 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:42,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,186 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:42,186 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:42,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:42,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:42,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:42,186 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:42,186 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:42,186 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,186 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:42,186 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:42,186 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,186 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:42,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:42,186 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,187 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,187 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:42,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195342269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,286 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:42,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:42,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:42,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:42,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:42,289 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:42,290 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:42,290 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:42,290 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:42,290 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,290 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,291 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,291 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:42,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:42,292 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,292 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:42,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:42,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:42,293 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:42,293 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:42,293 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:42,293 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:42,293 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:42,293 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:42,293 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:42,293 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:42,293 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:42,293 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:42,293 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:42,293 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,293 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:42,293 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:42,295 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,295 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:42,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:42,295 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,295 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:42,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:42,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:42,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50411598[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:42,297 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:42,297 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,298 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,298 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:42,298 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:42,298 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:42,298 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,298 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:42,300 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,300 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:42,300 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:42,301 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:42,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:42,303 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:42,303 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:42,303 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:42,303 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:42,303 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:42,305 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,305 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:42,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,305 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:42,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:42,306 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:42,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,307 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:42,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:42,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34414 deadline: 1680195342307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,310 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,310 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:42,310 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:42,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:42,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:42,310 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:42,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:42,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:42,310 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,310 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:42,310 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,310 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,310 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,310 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,310 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,310 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:42,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:42,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:42,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:42,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34482 deadline: 1680195342308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:42,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34442 deadline: 1680195342313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,410 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:42,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:42,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:42,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:42,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:42,413 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:42,413 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:42,413 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:42,413 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:42,414 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,414 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,415 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:42,415 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:42,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:42,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,415 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,416 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:42,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:42,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:42,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:42,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:42,417 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:42,417 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:42,417 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:42,417 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:42,417 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:42,417 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:42,417 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:42,417 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:42,417 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:42,417 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:42,417 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:42,417 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,417 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:42,417 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:42,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,420 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:42,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:42,420 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,420 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:42,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:42,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:42,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:42,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:42,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c8488a7[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:42,421 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:42,421 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,425 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,425 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,425 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:42,425 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:42,425 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:42,425 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:42,426 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:42,426 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:42,426 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,427 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:42,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:42,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:42,429 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:42,429 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:42,429 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:42,429 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:42,429 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:42,430 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,430 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:42,430 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:42,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:42,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:42,431 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:42,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,432 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:42,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,435 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:42,435 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:42,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:42,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:42,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:42,435 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,435 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:42,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:42,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:42,435 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,435 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:42,435 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:42,435 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,435 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,435 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,436 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,436 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:42,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:42,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:42,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34456 deadline: 1680195342527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,536 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:42,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:42,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:42,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:42,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:42,539 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:42,539 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:42,539 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:42,539 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:42,539 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,539 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,541 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,541 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,541 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:42,542 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:42,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:42,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:42,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:42,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:42,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:42,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:42,543 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:42,543 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:42,543 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:42,544 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:42,544 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:42,544 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:42,544 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:42,544 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:42,544 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:42,544 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:42,544 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:42,544 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,544 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:42,544 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:42,544 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/95c7666bbdaa4061a26b21a0bc55ccd6 2023-03-30 16:54:42,546 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,546 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:42,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:42,546 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,546 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:42,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:42,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:42,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:42,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:42,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ab43494[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:42,548 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:42,548 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,549 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,549 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:42,549 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,549 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:42,549 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:42,549 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:42,552 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,552 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:42,552 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:42,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:42,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:42,552 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:42,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:42,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:42,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,555 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/59206a349e2b4fedb10edb34456e3493 2023-03-30 16:54:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:42,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:42,560 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:42,560 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:42,560 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:42,560 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:42,560 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:42,562 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/b8f1d38a233646e98a4e50d3569b9194 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/b8f1d38a233646e98a4e50d3569b9194 2023-03-30 16:54:42,562 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,563 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:42,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:42,563 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:42,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:42,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:42,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:42,563 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:42,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,565 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:42,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:42,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,567 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/b8f1d38a233646e98a4e50d3569b9194, entries=200, sequenceid=210, filesize=14.1 K 2023-03-30 16:54:42,567 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/95c7666bbdaa4061a26b21a0bc55ccd6 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/95c7666bbdaa4061a26b21a0bc55ccd6 2023-03-30 16:54:42,568 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:42,568 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:42,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:42,568 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:42,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:42,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:42,568 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,568 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:42,568 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,569 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,569 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,569 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:42,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:42,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:42,569 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,569 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:42,569 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:42,571 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/95c7666bbdaa4061a26b21a0bc55ccd6, entries=150, sequenceid=210, filesize=11.8 K 2023-03-30 16:54:42,572 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/59206a349e2b4fedb10edb34456e3493 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/59206a349e2b4fedb10edb34456e3493 2023-03-30 16:54:42,575 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/59206a349e2b4fedb10edb34456e3493, entries=150, sequenceid=210, filesize=11.8 K 2023-03-30 16:54:42,576 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 2720c59305e818889603edf8ccad1c2c in 950ms, sequenceid=210, compaction requested=false 2023-03-30 16:54:42,576 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:42,669 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:42,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:42,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:42,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:42,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:42,673 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:42,673 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:42,673 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:42,673 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:42,674 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,674 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,675 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:42,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:42,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:42,675 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,676 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,676 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:42,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:42,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:42,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:42,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:42,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:42,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:42,677 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:42,677 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:42,677 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:42,677 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:42,677 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:42,677 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:42,677 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:42,677 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-03-30 16:54:42,678 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:42,678 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:42,678 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:42,678 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:42,678 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:42,678 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:42,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:42,776 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:42,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:42,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195342805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:42,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34414 deadline: 1680195342812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:42,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34482 deadline: 1680195342816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:42,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34442 deadline: 1680195342818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:42,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:42,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195342910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,090 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/d9165028db664ef1a695a23369089a2b 2023-03-30 16:54:43,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:43,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195343115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:43,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195343418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,501 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/23c8e1b6f29141df9949ff18222b7ccb 2023-03-30 16:54:43,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:43,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34456 deadline: 1680195343539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:43,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34414 deadline: 1680195343818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:43,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34442 deadline: 1680195343821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:43,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34482 deadline: 1680195343829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,920 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/1babef462c554755a8b62fb79c59ebe5 2023-03-30 16:54:43,925 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/d9165028db664ef1a695a23369089a2b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/d9165028db664ef1a695a23369089a2b 2023-03-30 16:54:43,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:43,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195343925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,931 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/d9165028db664ef1a695a23369089a2b, entries=150, sequenceid=246, filesize=11.8 K 2023-03-30 16:54:43,932 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/23c8e1b6f29141df9949ff18222b7ccb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/23c8e1b6f29141df9949ff18222b7ccb 2023-03-30 16:54:43,937 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/23c8e1b6f29141df9949ff18222b7ccb, entries=150, sequenceid=246, filesize=11.8 K 2023-03-30 16:54:43,937 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/1babef462c554755a8b62fb79c59ebe5 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/1babef462c554755a8b62fb79c59ebe5 2023-03-30 16:54:43,945 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/1babef462c554755a8b62fb79c59ebe5, entries=150, sequenceid=246, filesize=11.8 K 2023-03-30 16:54:43,946 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for 2720c59305e818889603edf8ccad1c2c in 1269ms, sequenceid=246, compaction requested=true 2023-03-30 16:54:43,946 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:43,946 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:43,946 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:43,946 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:43,946 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,946 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:43,946 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:43,949 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:43,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:43,949 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:43,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:43,949 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:43,949 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:43,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:43,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:43,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:43,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:43,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27674b8d[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:43,950 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:43,950 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:43,952 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:43,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:43,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:43,952 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:43,952 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:43,952 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:43,952 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,952 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:43,953 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,953 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:43,953 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:43,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:43,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:43,954 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:43,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:43,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:43,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:43,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:43,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:43,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:43,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:43,958 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:43,958 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:43,958 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:43,958 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:43,958 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:43,960 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:43,960 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:43,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:43,960 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:43,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:43,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:43,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:43,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:43,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:43,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:43,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:43,962 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:43,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:43,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:43,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,964 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:43,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:43,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:43,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:43,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:43,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:43,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:43,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:43,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:43,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:43,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:43,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:43,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:43,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:43,978 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:43,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:43,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:43,978 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:43,978 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:44,078 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:44,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:44,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:44,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:44,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:44,084 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:44,084 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:44,084 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:44,084 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:44,085 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,085 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:44,086 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:44,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:44,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:44,086 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:44,087 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,087 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:44,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:44,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:44,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:44,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:44,092 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:44,092 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:44,092 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:44,104 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:44,104 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:44,104 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:44,104 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:44,104 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-30 16:54:44,104 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:44,104 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:44,104 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:44,105 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:44,105 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:44,105 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:44,522 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/da89c421a2ba41ba9a1b856f1d904f91 2023-03-30 16:54:44,534 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/bb65618f8ff5427f9ca9afe61087cb79 2023-03-30 16:54:44,545 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/87a67fb20c2245f1ad933ec7b8bdbb47 2023-03-30 16:54:44,549 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/da89c421a2ba41ba9a1b856f1d904f91 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/da89c421a2ba41ba9a1b856f1d904f91 2023-03-30 16:54:44,552 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/da89c421a2ba41ba9a1b856f1d904f91, entries=150, sequenceid=258, filesize=11.8 K 2023-03-30 16:54:44,553 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/bb65618f8ff5427f9ca9afe61087cb79 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/bb65618f8ff5427f9ca9afe61087cb79 2023-03-30 16:54:44,556 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/bb65618f8ff5427f9ca9afe61087cb79, entries=150, sequenceid=258, filesize=11.8 K 2023-03-30 16:54:44,557 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/87a67fb20c2245f1ad933ec7b8bdbb47 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/87a67fb20c2245f1ad933ec7b8bdbb47 2023-03-30 16:54:44,560 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/87a67fb20c2245f1ad933ec7b8bdbb47, entries=150, sequenceid=258, filesize=11.8 K 2023-03-30 16:54:44,561 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 2720c59305e818889603edf8ccad1c2c in 457ms, sequenceid=258, compaction requested=true 2023-03-30 16:54:44,561 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:44,561 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:44,561 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:44,561 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:44,561 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,561 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:44,561 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:44,563 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,563 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:44,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:44,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:44,563 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,563 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:44,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:44,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:44,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:44,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:44,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fd2772c[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:44,565 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:44,565 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,566 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,567 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:44,567 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:44,567 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:44,567 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,567 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:44,572 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,572 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:44,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:44,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:44,572 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:44,572 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:44,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:44,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:44,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:44,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:44,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:44,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:44,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:44,574 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:44,574 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:44,574 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:44,574 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:44,575 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:44,577 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,577 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,577 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:44,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,578 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,582 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,584 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,584 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:44,584 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:44,584 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:44,584 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:44,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:44,584 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:44,584 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,584 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:44,584 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,584 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:44,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:44,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:44,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:44,585 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:44,585 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:44,685 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:44,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:44,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:44,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:44,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:44,691 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:44,691 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:44,691 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:44,691 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:44,691 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,691 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:44,693 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,693 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:44,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:44,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:44,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:44,694 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,694 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:44,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:44,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:44,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:44,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:44,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:44,695 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:44,695 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:44,695 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:44,695 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:44,695 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:44,695 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:44,696 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:44,696 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:44,696 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:44,696 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:44,696 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:44,696 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,696 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:44,696 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:44,698 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,699 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:44,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:44,699 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,699 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:44,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:44,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:44,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:44,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@274b09c1[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:44,702 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:44,702 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,703 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,704 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:44,704 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:44,704 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:44,704 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,704 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:44,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:44,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:44,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:44,706 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:44,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:44,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:44,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:44,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:44,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:44,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:44,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:44,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:44,709 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:44,709 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:44,709 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:44,709 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:44,709 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:44,711 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,711 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,711 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:44,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,711 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:44,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:44,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:44,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:44,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:44,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:44,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:44,713 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:44,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:44,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:44,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,717 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,718 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:44,718 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:44,718 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,718 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:44,718 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:44,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:44,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:44,718 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,718 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,718 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,718 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:44,718 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:44,718 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:44,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:44,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:44,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:44,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:44,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:44,818 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:44,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:44,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:44,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:44,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:44,826 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:44,826 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:44,826 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:44,826 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:44,826 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,826 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:44,828 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:44,828 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:44,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:44,828 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,828 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:44,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:44,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:44,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:44,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:44,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:44,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:44,830 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:44,830 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:44,830 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:44,831 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:44,831 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:44,831 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:44,831 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:44,831 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:44,831 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:44,831 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:44,831 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:44,831 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,831 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:44,831 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:44,835 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,835 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:44,840 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,840 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:44,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:44,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:44,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:44,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cd08928[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:44,845 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:44,845 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,847 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,847 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:44,847 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,847 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:44,847 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:44,847 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:44,850 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:44,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:44,850 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:44,850 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:44,850 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:44,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:44,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:44,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:44,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:44,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:44,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:44,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:44,855 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:44,855 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:44,855 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:44,855 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:44,855 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:44,857 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,857 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:44,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,857 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:44,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:44,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:44,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:44,857 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:44,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:44,859 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:44,862 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:44,862 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:44,862 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:44,862 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:44,862 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:44,862 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:44,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:44,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:44,862 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,863 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,863 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:44,863 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,863 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:44,863 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:44,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:44,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:44,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:44,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:44,943 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:54:44,943 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:44,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:44,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:44,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:44,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:44,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:44,958 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/4751def338304e618029d8131df531fb 2023-03-30 16:54:44,963 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:44,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:44,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:44,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:44,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:44,992 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:44,992 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:44,992 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:44,992 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:44,992 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,993 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:44,994 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,994 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:44,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:44,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:44,994 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:44,994 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:44,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:44,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:45,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:45,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:45,001 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:45,001 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:45,001 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:45,004 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:45,004 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:45,004 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:45,004 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:45,004 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:45,004 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:45,004 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:45,004 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:45,004 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195284987, max timestamp=1680195285013Num uniques 859; 2023-03-30 16:54:45,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195284987, max timestamp=1680195285013Num uniques 859; 2023-03-30 16:54:45,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:45,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195284987, max timestamp=1680195285013Num uniques 859; 2023-03-30 16:54:45,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:45,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:45,016 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:45,016 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:45,018 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/ed6b59760cf64407b6e48f96a19af830 2023-03-30 16:54:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,018 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,019 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,019 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:45,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31f6841c[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:45,021 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:45,021 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,023 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,023 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:45,023 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:45,023 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:45,025 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:45,025 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:45,025 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:45,025 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,026 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:45,026 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:45,026 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:45,026 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:45,026 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:45,026 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:45,026 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:45,028 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,028 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,028 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:45,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:45,029 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,030 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,035 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,035 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,035 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,036 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:45,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:45,035 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:45,036 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:45,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:45,036 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/7457f10778524d9fb0f88e4fe996d4c7 2023-03-30 16:54:45,036 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,036 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,036 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,036 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:45,036 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:45,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:45,036 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:45,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:45,041 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/4751def338304e618029d8131df531fb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/4751def338304e618029d8131df531fb 2023-03-30 16:54:45,045 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/4751def338304e618029d8131df531fb, entries=150, sequenceid=270, filesize=11.9 K 2023-03-30 16:54:45,046 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/ed6b59760cf64407b6e48f96a19af830 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/ed6b59760cf64407b6e48f96a19af830 2023-03-30 16:54:45,050 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/ed6b59760cf64407b6e48f96a19af830, entries=150, sequenceid=270, filesize=11.9 K 2023-03-30 16:54:45,051 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/7457f10778524d9fb0f88e4fe996d4c7 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/7457f10778524d9fb0f88e4fe996d4c7 2023-03-30 16:54:45,054 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/7457f10778524d9fb0f88e4fe996d4c7, entries=150, sequenceid=270, filesize=11.9 K 2023-03-30 16:54:45,055 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=73.80 KB/75570 for 2720c59305e818889603edf8ccad1c2c in 112ms, sequenceid=270, compaction requested=true 2023-03-30 16:54:45,055 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:45,055 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:45,055 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 16:54:45,055 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 16:54:45,055 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:45,055 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:45,057 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60620 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 16:54:45,057 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 63010 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 16:54:45,057 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/B is initiating minor compaction (all files) 2023-03-30 16:54:45,057 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/A is initiating minor compaction (all files) 2023-03-30 16:54:45,057 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/B in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:45,057 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/A in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:45,057 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/62c78c882a27414f9e000303debb4a3e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/95c7666bbdaa4061a26b21a0bc55ccd6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/23c8e1b6f29141df9949ff18222b7ccb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/bb65618f8ff5427f9ca9afe61087cb79, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/ed6b59760cf64407b6e48f96a19af830] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=59.2 K 2023-03-30 16:54:45,057 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/1c29b57a95d945f9a7e3a25f2be4cb49, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/b8f1d38a233646e98a4e50d3569b9194, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/d9165028db664ef1a695a23369089a2b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/da89c421a2ba41ba9a1b856f1d904f91, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/4751def338304e618029d8131df531fb] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=61.5 K 2023-03-30 16:54:45,058 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 62c78c882a27414f9e000303debb4a3e, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680195281312 2023-03-30 16:54:45,058 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 1c29b57a95d945f9a7e3a25f2be4cb49, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680195281312 2023-03-30 16:54:45,058 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 95c7666bbdaa4061a26b21a0bc55ccd6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1680195281399 2023-03-30 16:54:45,058 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting b8f1d38a233646e98a4e50d3569b9194, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1680195281367 2023-03-30 16:54:45,059 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 23c8e1b6f29141df9949ff18222b7ccb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680195281690 2023-03-30 16:54:45,059 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting d9165028db664ef1a695a23369089a2b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680195281690 2023-03-30 16:54:45,059 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting bb65618f8ff5427f9ca9afe61087cb79, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1680195282788 2023-03-30 16:54:45,059 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting da89c421a2ba41ba9a1b856f1d904f91, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1680195282788 2023-03-30 16:54:45,060 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting ed6b59760cf64407b6e48f96a19af830, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680195284935 2023-03-30 16:54:45,060 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 4751def338304e618029d8131df531fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680195284935 2023-03-30 16:54:45,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:45,065 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=80.51 KB heapSize=225.61 KB 2023-03-30 16:54:45,065 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:45,065 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:45,065 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:45,065 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:45,065 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:45,065 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:45,071 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#B#compaction#211 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:45,072 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#A#compaction#212 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:45,086 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/24ba73a3dddf42dd89a691db0bedcfc0 2023-03-30 16:54:45,094 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/4b1345e537df442faa248733acc473ed as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/4b1345e537df442faa248733acc473ed 2023-03-30 16:54:45,102 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/B of 2720c59305e818889603edf8ccad1c2c into 4b1345e537df442faa248733acc473ed(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:45,102 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:45,102 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/B, priority=11, startTime=1680195285055; duration=0sec 2023-03-30 16:54:45,103 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:45,103 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 16:54:45,106 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60620 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 16:54:45,106 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/C is initiating minor compaction (all files) 2023-03-30 16:54:45,106 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/C in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:45,106 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/3a85f5f248e54cf08b709d9b1647f542, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/59206a349e2b4fedb10edb34456e3493, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/1babef462c554755a8b62fb79c59ebe5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/87a67fb20c2245f1ad933ec7b8bdbb47, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/7457f10778524d9fb0f88e4fe996d4c7] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=59.2 K 2023-03-30 16:54:45,106 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 3a85f5f248e54cf08b709d9b1647f542, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680195281312 2023-03-30 16:54:45,107 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 59206a349e2b4fedb10edb34456e3493, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1680195281399 2023-03-30 16:54:45,107 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 1babef462c554755a8b62fb79c59ebe5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680195281690 2023-03-30 16:54:45,107 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 87a67fb20c2245f1ad933ec7b8bdbb47, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1680195282788 2023-03-30 16:54:45,108 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 7457f10778524d9fb0f88e4fe996d4c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680195284935 2023-03-30 16:54:45,122 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#C#compaction#214 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:45,137 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:45,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:45,138 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/cce8467d74d242a0863dd45913186f5a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/cce8467d74d242a0863dd45913186f5a 2023-03-30 16:54:45,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:45,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:45,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:45,141 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:45,141 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:45,141 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:45,141 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:45,141 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,141 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,143 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:45,143 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:45,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:45,144 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,144 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,144 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/C of 2720c59305e818889603edf8ccad1c2c into cce8467d74d242a0863dd45913186f5a(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:45,144 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:45,144 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/C, priority=11, startTime=1680195285055; duration=0sec 2023-03-30 16:54:45,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,144 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:45,145 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:45,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:45,145 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:45,145 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:45,145 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:45,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:45,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:45,146 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:45,146 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:45,146 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:45,146 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:45,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:45,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:45,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:45,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:45,148 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,148 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,149 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,149 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:45,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d71ee19[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:45,150 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:45,150 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:45,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195345146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,152 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,152 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:45,152 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:45,152 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:45,156 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,156 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:45,156 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:45,157 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:45,157 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:45,157 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:45,157 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:45,157 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:45,158 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,158 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,159 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,159 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,160 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,162 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,162 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:45,162 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,163 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:45,163 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:45,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:45,163 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:45,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:45,163 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,163 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,163 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,163 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:45,163 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:45,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:45,163 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:45,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:45,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195345253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,263 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:45,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:45,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:45,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:45,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:45,266 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:45,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:45,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:45,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:45,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,268 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:45,268 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:45,269 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,269 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:45,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:45,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:45,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:45,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:45,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:45,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:45,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:45,270 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:45,270 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:45,270 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:45,270 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:45,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:45,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:45,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:45,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:45,273 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,273 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,273 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,273 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:45,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@220be177[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:45,275 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:45,275 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,276 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,276 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:45,276 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:45,276 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:45,276 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,276 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:45,277 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:45,277 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,277 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:45,277 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:45,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:45,280 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:45,280 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:45,280 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:45,280 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:45,280 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:45,281 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,281 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,281 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:45,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,282 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,283 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,286 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,286 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,286 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:45,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:45,286 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:45,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:45,286 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,286 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:45,286 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,286 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:45,286 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,286 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:45,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:45,286 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:45,286 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:45,387 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:45,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:45,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:45,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:45,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:45,390 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:45,390 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:45,390 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:45,390 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:45,390 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,390 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,392 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:45,392 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:45,392 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,392 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:45,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:45,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:45,394 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:45,394 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:45,394 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:45,394 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:45,394 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:45,394 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:45,394 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:45,394 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:45,394 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,394 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:45,394 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:45,396 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,396 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,397 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,397 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:45,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:45,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@abab5d7[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:45,398 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:45,398 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,399 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,399 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:45,399 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:45,399 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:45,399 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,399 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:45,406 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,406 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,406 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:45,406 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:45,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:45,408 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:45,408 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:45,408 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:45,408 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:45,408 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:45,410 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,410 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:45,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,410 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,410 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,411 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,416 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:45,416 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:45,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:45,416 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,416 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,416 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,416 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,416 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,416 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,416 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:45,416 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:45,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:45,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:45,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:45,416 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:45,417 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:45,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:45,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:45,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195345456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,496 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/bc80ad3a029748128b606cb83f37eca4 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/bc80ad3a029748128b606cb83f37eca4 2023-03-30 16:54:45,499 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/ee571cc789d643e5b2b08d6198626f99 2023-03-30 16:54:45,501 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/A of 2720c59305e818889603edf8ccad1c2c into bc80ad3a029748128b606cb83f37eca4(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:45,502 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:45,502 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/A, priority=11, startTime=1680195285055; duration=0sec 2023-03-30 16:54:45,502 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:45,517 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:45,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:45,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:45,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:45,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:45,521 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:45,521 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:45,521 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:45,521 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:45,521 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,521 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,523 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,523 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:45,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:45,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:45,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:45,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:45,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:45,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:45,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:45,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:45,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:45,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:45,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:45,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:45,525 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:45,525 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:45,525 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:45,525 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:45,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:45,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:45,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:45,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:45,527 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,527 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,527 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,527 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:45,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f6906f8[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:45,529 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:45,529 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,530 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,530 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:45,530 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:45,530 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:45,530 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,530 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:45,532 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,532 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,532 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:45,532 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:45,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:45,534 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:45,534 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:45,534 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:45,534 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:45,535 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:45,541 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,541 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:45,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,542 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,543 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,545 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,547 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,547 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,547 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,547 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,547 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,547 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:45,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:45,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:45,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:45,547 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,547 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,547 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:45,548 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:45,548 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:45,548 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:45,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:45,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:45,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:45,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34456 deadline: 1680195345555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,559 DEBUG [Thread-1451] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:54:45,648 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:45,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:45,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:45,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:45,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:45,651 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:45,651 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:45,651 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:45,651 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:45,651 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,651 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,652 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:45,652 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:45,653 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,653 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:45,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:45,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:45,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:45,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:45,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:45,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:45,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:45,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:45,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:45,654 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:45,654 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:45,654 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:45,654 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:45,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:45,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:45,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:45,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:45,656 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,656 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,656 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,656 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:45,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dba202a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:45,657 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:45,657 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,660 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,660 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,660 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:45,660 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:45,660 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:45,660 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:45,661 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,661 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:45,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,661 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:45,661 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:45,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:45,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:45,663 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:45,664 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:45,664 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:45,664 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:45,664 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:45,665 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,665 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,665 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:45,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,666 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:45,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:45,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,667 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,669 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,669 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:45,669 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,669 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:45,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:45,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:45,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:45,669 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:45,669 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:45,669 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,669 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:45,670 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,670 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,670 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:45,670 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:45,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:45,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34430 deadline: 1680195345761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,770 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:45,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:45,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:45,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:45,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:45,773 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:45,773 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:45,773 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:45,773 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:45,773 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,773 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,776 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:45,776 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:45,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:45,776 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,776 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:45,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:45,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:45,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:45,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:45,777 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:45,777 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:45,777 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:45,778 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:45,778 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:45,778 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:45,778 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:45,778 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:45,778 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:45,778 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:45,778 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:45,778 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,778 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:45,778 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:45,779 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,780 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,780 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,780 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:45,781 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:45,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20c762b3[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:45,781 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,782 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,783 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:45,783 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,783 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:45,783 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:45,783 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:45,784 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,784 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:45,784 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:45,784 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:45,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:45,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:45,787 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:45,787 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:45,787 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:45,787 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:45,787 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:45,791 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,791 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,791 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:45,791 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,793 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,794 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,794 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,795 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,795 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:45,795 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,795 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:45,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:45,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:45,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:45,795 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:45,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:45,795 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:45,795 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:45,795 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,795 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:45,795 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:45,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:45,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:45,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34442 deadline: 1680195345832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,835 DEBUG [Thread-1447] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:54:45,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:45,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34414 deadline: 1680195345835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,838 DEBUG [Thread-1443] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:54:45,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:45,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34482 deadline: 1680195345845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,849 DEBUG [Thread-1445] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:54:45,895 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:45,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:45,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:45,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:45,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:45,898 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:45,898 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:45,898 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:45,898 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:45,899 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,899 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,903 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,903 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:45,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:45,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:45,903 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,903 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:45,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:45,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:45,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:45,904 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:45,904 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:45,904 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:45,904 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:45,904 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:45,904 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:45,904 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:45,904 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:45,904 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:45,904 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:45,905 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:45,905 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,905 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:45,905 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:45,910 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,910 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,910 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,910 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:45,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:45,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f0c6f71[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:45,911 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:45,911 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,913 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,913 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:45,913 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,913 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:45,913 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:45,913 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:45,914 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/7777c4885d3c473887073a86e17a4388 2023-03-30 16:54:45,914 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,915 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:45,915 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,915 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:45,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:45,917 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:45,917 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:45,917 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:45,917 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:45,917 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:45,919 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/24ba73a3dddf42dd89a691db0bedcfc0 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/24ba73a3dddf42dd89a691db0bedcfc0 2023-03-30 16:54:45,920 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,920 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,920 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:45,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:45,920 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:45,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:45,921 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:45,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:45,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,925 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,925 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:45,925 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:45,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:45,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:45,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:45,925 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:45,925 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:45,925 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:45,925 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:45,925 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:45,926 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:45,925 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,926 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:45,926 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:45,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:45,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:45,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:45,930 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/24ba73a3dddf42dd89a691db0bedcfc0, entries=200, sequenceid=299, filesize=14.3 K 2023-03-30 16:54:45,930 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/ee571cc789d643e5b2b08d6198626f99 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/ee571cc789d643e5b2b08d6198626f99 2023-03-30 16:54:45,934 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/ee571cc789d643e5b2b08d6198626f99, entries=150, sequenceid=299, filesize=11.9 K 2023-03-30 16:54:45,934 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/7777c4885d3c473887073a86e17a4388 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/7777c4885d3c473887073a86e17a4388 2023-03-30 16:54:45,938 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/7777c4885d3c473887073a86e17a4388, entries=150, sequenceid=299, filesize=11.9 K 2023-03-30 16:54:45,939 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~225.56 KB/230976, currentSize=100.63 KB/103050 for 2720c59305e818889603edf8ccad1c2c in 873ms, sequenceid=299, compaction requested=false 2023-03-30 16:54:45,939 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:46,026 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:46,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:46,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:46,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:46,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:46,029 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:46,029 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:46,029 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:46,029 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:46,030 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,030 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,031 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:46,031 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:46,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:46,031 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,031 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:46,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:46,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:46,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:46,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:46,033 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:46,033 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:46,033 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:46,033 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:46,033 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:46,033 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:46,033 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:46,034 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-30 16:54:46,034 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:46,034 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:46,034 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:46,034 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:46,034 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:46,034 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:46,054 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/fc53863e0b6a467d9ba1ed3c8fa591d8 2023-03-30 16:54:46,071 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/6769e481c46f4ec18ec2e84dc17beb86 2023-03-30 16:54:46,092 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/3be67ce6778d4a37934e3a5d195361bf 2023-03-30 16:54:46,097 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/fc53863e0b6a467d9ba1ed3c8fa591d8 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/fc53863e0b6a467d9ba1ed3c8fa591d8 2023-03-30 16:54:46,101 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/fc53863e0b6a467d9ba1ed3c8fa591d8, entries=150, sequenceid=320, filesize=11.9 K 2023-03-30 16:54:46,103 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/6769e481c46f4ec18ec2e84dc17beb86 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/6769e481c46f4ec18ec2e84dc17beb86 2023-03-30 16:54:46,107 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/6769e481c46f4ec18ec2e84dc17beb86, entries=150, sequenceid=320, filesize=11.9 K 2023-03-30 16:54:46,108 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/3be67ce6778d4a37934e3a5d195361bf as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/3be67ce6778d4a37934e3a5d195361bf 2023-03-30 16:54:46,117 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/3be67ce6778d4a37934e3a5d195361bf, entries=150, sequenceid=320, filesize=11.9 K 2023-03-30 16:54:46,118 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 2720c59305e818889603edf8ccad1c2c in 84ms, sequenceid=320, compaction requested=true 2023-03-30 16:54:46,118 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:46,118 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:46,118 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:46,118 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:46,118 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,118 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:46,118 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:46,120 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,120 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,120 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,120 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:46,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3815f8ae[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:46,122 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:46,122 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,123 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,123 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:46,123 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:46,123 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:46,123 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,124 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:46,126 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,126 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:46,126 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,126 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:46,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:46,132 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:46,132 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:46,132 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:46,132 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:46,132 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:46,135 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,135 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:46,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,136 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,137 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,138 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,140 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,140 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:46,140 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:46,140 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:46,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:46,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:46,140 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:46,140 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,141 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,141 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,141 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,141 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,141 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:46,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:46,141 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:46,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:46,241 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:46,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:46,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:46,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:46,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:46,245 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:46,245 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:46,245 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:46,245 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:46,246 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,246 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,247 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:46,247 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:46,248 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,248 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:46,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:46,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:46,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:46,249 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:46,249 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:46,249 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:46,250 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:46,250 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:46,250 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:46,250 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:46,250 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:46,250 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:46,250 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:46,250 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:46,250 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,250 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:46,250 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:46,252 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,253 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,253 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,253 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:46,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34a234f6[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:46,254 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:46,254 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,256 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,256 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:46,256 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:46,256 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:46,256 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,256 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:46,257 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,257 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:46,257 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,257 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:46,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:46,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:46,260 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:46,260 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:46,260 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:46,260 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:46,260 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:46,262 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,262 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:46,262 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,263 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,264 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,268 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,268 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,268 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:46,268 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:46,268 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:46,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:46,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:46,268 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,268 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:46,268 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,268 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,268 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,268 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:46,268 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:46,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:46,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:46,282 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:54:46,282 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:46,282 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:46,282 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:46,282 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:46,282 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:46,282 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:46,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195286312, max timestamp=1680195286334Num uniques 859; 2023-03-30 16:54:46,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195286312, max timestamp=1680195286334Num uniques 859; 2023-03-30 16:54:46,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195286312, max timestamp=1680195286334Num uniques 859; 2023-03-30 16:54:46,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:46,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:46,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:46,369 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:46,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:46,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:46,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:46,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:46,372 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:46,372 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:46,372 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:46,372 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:46,372 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,372 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,373 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,373 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:46,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:46,374 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,374 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:46,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:46,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:46,375 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:46,375 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:46,375 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:46,375 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:46,375 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:46,375 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:46,375 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:46,375 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:46,375 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:46,375 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:46,375 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:46,375 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,375 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:46,375 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:46,377 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,377 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,377 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,377 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:46,378 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:46,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2feb1a2b[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:46,378 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,381 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,381 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:46,381 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:46,381 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:46,381 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,381 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:46,383 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,383 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:46,383 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:46,383 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:46,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:46,386 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:46,386 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:46,386 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:46,386 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:46,386 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:46,388 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,388 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:46,388 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,388 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,390 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,392 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:46,392 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:46,392 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,392 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,392 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,392 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,392 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,392 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,392 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:46,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:46,392 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:46,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:46,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:46,393 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:46,393 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:46,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:46,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:46,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195346399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,493 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:46,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:46,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:46,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:46,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:46,496 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:46,496 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:46,496 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:46,496 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:46,496 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,496 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,498 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:46,499 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:46,499 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,499 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:46,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:46,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:46,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:46,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:46,500 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:46,500 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:46,500 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:46,500 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:46,500 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:46,502 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,502 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,502 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,502 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:46,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33809f40[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:46,504 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:46,504 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,505 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,505 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:46,505 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:46,505 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,505 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:46,505 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:46,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:46,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195346503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,506 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:46,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:46,509 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:46,509 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:46,509 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:46,509 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:46,509 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:46,511 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,511 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,511 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:46,512 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,513 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,515 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,515 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,515 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:46,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:46,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:46,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:46,515 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,515 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:46,515 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:46,515 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,515 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:46,516 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:46,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:46,515 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,516 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,516 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:46,616 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:46,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:46,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:46,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:46,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:46,621 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:46,621 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:46,621 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:46,621 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:46,621 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,621 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:46,622 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:46,623 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,623 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:46,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:46,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:46,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:46,624 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:46,624 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:46,624 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:46,624 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:46,624 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:46,624 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:46,624 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:46,624 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:46,624 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:46,624 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:46,624 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:46,625 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,625 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:46,625 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:46,626 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,626 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,626 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,626 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:46,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24c7d116[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:46,627 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:46,627 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,629 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,629 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:46,629 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:46,630 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:46,630 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,630 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:46,631 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:46,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:46,633 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:46,633 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:46,633 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:46,633 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:46,633 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:46,635 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,635 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,635 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:46,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,635 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,637 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,639 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,639 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,639 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,639 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:46,639 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:46,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:46,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:46,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:46,639 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:46,639 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:46,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:46,639 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:46,639 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,640 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:46,640 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,640 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:46,690 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/2238984969604415bfeea5193e12cc2c 2023-03-30 16:54:46,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:46,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195346716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,722 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/26122dd9c9e44f3bbe5da85a2b6bf639 2023-03-30 16:54:46,740 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:46,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:46,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:46,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:46,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:46,750 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:46,750 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:46,750 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:46,750 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:46,750 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,750 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,752 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:46,752 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:46,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:46,753 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,753 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:46,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:46,754 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:46,754 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:46,754 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:46,756 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:46,756 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:46,756 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:46,756 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:46,758 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,758 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,758 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,758 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:46,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e754fc[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:46,760 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:46,760 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,761 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,761 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:46,762 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:46,762 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:46,762 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,762 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:46,767 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:46,767 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:46,767 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,767 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:46,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:46,769 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:46,769 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:46,769 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:46,769 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:46,769 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:46,771 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,771 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,771 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:46,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,772 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,773 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,776 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,776 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:46,776 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:46,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:46,776 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:46,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:46,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:46,776 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:46,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:46,776 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:46,776 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:46,776 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:46,777 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,777 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,777 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,777 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,876 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:46,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:46,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:46,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:46,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:46,880 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:46,880 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:46,880 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:46,880 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:46,880 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,880 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,882 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,882 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:46,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:46,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:46,882 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,882 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:46,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:46,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:46,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:46,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:46,883 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:46,883 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:46,883 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:46,883 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:46,883 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:46,883 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:46,884 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:46,884 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:46,884 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:46,886 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,886 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,886 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,886 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:46,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@762a2ca9[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:46,888 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:46,888 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,889 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,889 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:46,889 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:46,889 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:46,889 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,889 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:46,890 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,890 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:46,891 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:46,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,891 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:46,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:46,893 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:46,893 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:46,893 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:46,893 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:46,893 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:46,894 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:46,895 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,895 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:46,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:46,895 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,896 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:46,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:46,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:46,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:46,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:46,899 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:46,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,900 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:46,900 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:46,900 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,900 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:46,900 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:46,900 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:46,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:46,900 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:46,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:46,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:46,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:46,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:47,000 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:47,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:47,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:47,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:47,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:47,003 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:47,003 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:47,003 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:47,003 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:47,004 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:47,004 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:47,005 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:47,005 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:47,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:47,005 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,005 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:47,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:47,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:47,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:47,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:47,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:47,006 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:47,006 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:47,007 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:47,007 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:47,007 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:47,007 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:47,009 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,009 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:47,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:47,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:47,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:47,009 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:47,009 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:47,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:47,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:47,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:47,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:47,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41e7d7b3[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:47,011 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:47,011 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:47,012 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:47,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:47,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:47,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:47,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:47,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:47,012 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,012 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:47,019 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,019 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:47,019 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:47,019 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:47,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:47,021 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:47,021 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:47,021 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:47,022 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:47,022 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:47,023 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:47,023 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:47,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:47,023 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:47,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:47,024 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:47,025 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:47,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195347023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:47,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:47,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,027 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:47,027 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:47,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:47,027 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:47,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:47,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:47,027 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:47,027 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:47,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:47,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:47,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:47,028 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:47,028 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:47,028 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:47,028 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,028 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:47,028 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:47,028 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:47,128 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:47,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:47,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:47,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:47,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:47,131 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:47,131 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:47,131 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:47,131 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:47,131 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:47,132 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:47,133 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,133 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:47,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:47,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:47,133 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,133 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:47,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:47,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:47,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:47,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:47,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:47,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:47,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:47,134 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:47,134 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:47,134 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:47,134 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,134 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:47,135 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:47,139 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:47,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:47,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:47,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bb2f1f8[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:47,142 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:47,142 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:47,143 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:47,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:47,143 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:47,143 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:47,143 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:47,143 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,143 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:47,145 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:47,145 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:47,145 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:47,145 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:47,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:47,147 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:47,148 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:47,148 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:47,148 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:47,148 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:47,149 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:47,149 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:47,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:47,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:47,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:47,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:47,149 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:47,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:47,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:47,150 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:47,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:47,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:47,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:47,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:47,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:47,151 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:47,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:47,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,152 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/b0cf6884064e4d24a71e7938fc984a67 2023-03-30 16:54:47,156 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/2238984969604415bfeea5193e12cc2c as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/2238984969604415bfeea5193e12cc2c 2023-03-30 16:54:47,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:47,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:47,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:47,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:47,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:47,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:47,158 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:47,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:47,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:47,158 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:47,158 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:47,158 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:47,158 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:47,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:47,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:47,158 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:47,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:47,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/2238984969604415bfeea5193e12cc2c, entries=150, sequenceid=331, filesize=11.9 K 2023-03-30 16:54:47,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/26122dd9c9e44f3bbe5da85a2b6bf639 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/26122dd9c9e44f3bbe5da85a2b6bf639 2023-03-30 16:54:47,164 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/26122dd9c9e44f3bbe5da85a2b6bf639, entries=150, sequenceid=331, filesize=11.9 K 2023-03-30 16:54:47,165 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/b0cf6884064e4d24a71e7938fc984a67 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/b0cf6884064e4d24a71e7938fc984a67 2023-03-30 16:54:47,168 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/b0cf6884064e4d24a71e7938fc984a67, entries=150, sequenceid=331, filesize=11.9 K 2023-03-30 16:54:47,169 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2720c59305e818889603edf8ccad1c2c in 887ms, sequenceid=331, compaction requested=true 2023-03-30 16:54:47,169 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:47,169 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:47,169 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:54:47,169 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:54:47,169 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:47,169 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:47,171 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49154 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:54:47,171 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51594 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:54:47,171 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/B is initiating minor compaction (all files) 2023-03-30 16:54:47,171 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/A is initiating minor compaction (all files) 2023-03-30 16:54:47,171 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/B in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:47,171 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/A in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:47,171 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/4b1345e537df442faa248733acc473ed, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/ee571cc789d643e5b2b08d6198626f99, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/6769e481c46f4ec18ec2e84dc17beb86, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/26122dd9c9e44f3bbe5da85a2b6bf639] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=48.0 K 2023-03-30 16:54:47,171 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/bc80ad3a029748128b606cb83f37eca4, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/24ba73a3dddf42dd89a691db0bedcfc0, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/fc53863e0b6a467d9ba1ed3c8fa591d8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/2238984969604415bfeea5193e12cc2c] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=50.4 K 2023-03-30 16:54:47,171 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 4b1345e537df442faa248733acc473ed, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680195284935 2023-03-30 16:54:47,172 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting bc80ad3a029748128b606cb83f37eca4, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680195284935 2023-03-30 16:54:47,172 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting ee571cc789d643e5b2b08d6198626f99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680195285033 2023-03-30 16:54:47,172 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 24ba73a3dddf42dd89a691db0bedcfc0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680195285033 2023-03-30 16:54:47,172 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 6769e481c46f4ec18ec2e84dc17beb86, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680195285121 2023-03-30 16:54:47,172 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting fc53863e0b6a467d9ba1ed3c8fa591d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680195285121 2023-03-30 16:54:47,173 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 26122dd9c9e44f3bbe5da85a2b6bf639, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1680195286274 2023-03-30 16:54:47,173 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 2238984969604415bfeea5193e12cc2c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1680195286274 2023-03-30 16:54:47,190 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#B#compaction#222 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:47,194 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#A#compaction#223 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:47,208 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/0757e7cf841e4249ab09d837230af77d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/0757e7cf841e4249ab09d837230af77d 2023-03-30 16:54:47,214 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/A of 2720c59305e818889603edf8ccad1c2c into 0757e7cf841e4249ab09d837230af77d(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:47,214 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:47,214 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/A, priority=12, startTime=1680195287169; duration=0sec 2023-03-30 16:54:47,214 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:47,214 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:54:47,220 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49154 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:54:47,220 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/C is initiating minor compaction (all files) 2023-03-30 16:54:47,220 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/C in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:47,220 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/cce8467d74d242a0863dd45913186f5a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/7777c4885d3c473887073a86e17a4388, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/3be67ce6778d4a37934e3a5d195361bf, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/b0cf6884064e4d24a71e7938fc984a67] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=48.0 K 2023-03-30 16:54:47,226 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting cce8467d74d242a0863dd45913186f5a, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680195284935 2023-03-30 16:54:47,226 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 7777c4885d3c473887073a86e17a4388, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1680195285033 2023-03-30 16:54:47,227 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 3be67ce6778d4a37934e3a5d195361bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680195285121 2023-03-30 16:54:47,227 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting b0cf6884064e4d24a71e7938fc984a67, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1680195286274 2023-03-30 16:54:47,233 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#C#compaction#224 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:47,258 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:47,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:47,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:47,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:47,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:47,261 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:47,262 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:47,262 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:47,262 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:47,262 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:47,262 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:47,263 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:47,263 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:47,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:47,264 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,264 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:47,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:47,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:47,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:47,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:47,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:47,266 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:47,266 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:47,266 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:47,266 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:47,266 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:47,266 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:47,266 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:47,266 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-30 16:54:47,266 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:47,266 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:47,266 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:47,266 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:47,267 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:47,267 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:47,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:47,529 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:47,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:47,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195347547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,608 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/4f8a8ac4a5c849f592198486a398e066 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/4f8a8ac4a5c849f592198486a398e066 2023-03-30 16:54:47,612 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/B of 2720c59305e818889603edf8ccad1c2c into 4f8a8ac4a5c849f592198486a398e066(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:47,612 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:47,612 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/B, priority=12, startTime=1680195287169; duration=0sec 2023-03-30 16:54:47,612 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:47,646 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/d912324bc74b479ba8fa9d6bd2e5eb80 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/d912324bc74b479ba8fa9d6bd2e5eb80 2023-03-30 16:54:47,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:47,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195347650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:47,652 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/C of 2720c59305e818889603edf8ccad1c2c into d912324bc74b479ba8fa9d6bd2e5eb80(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:47,652 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:47,652 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/C, priority=12, startTime=1680195287169; duration=0sec 2023-03-30 16:54:47,652 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:47,681 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/85946b340f584ac28cbe3e23d06d0bde 2023-03-30 16:54:47,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:47,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195347852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,093 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/513d4cedfc074435b31740a80df5866e 2023-03-30 16:54:48,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:48,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195348157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,508 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/a9205a1bb7fc4c8685bc0a73e3337cc7 2023-03-30 16:54:48,514 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/85946b340f584ac28cbe3e23d06d0bde as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/85946b340f584ac28cbe3e23d06d0bde 2023-03-30 16:54:48,518 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/85946b340f584ac28cbe3e23d06d0bde, entries=150, sequenceid=369, filesize=11.9 K 2023-03-30 16:54:48,520 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/513d4cedfc074435b31740a80df5866e as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/513d4cedfc074435b31740a80df5866e 2023-03-30 16:54:48,529 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/513d4cedfc074435b31740a80df5866e, entries=150, sequenceid=369, filesize=11.9 K 2023-03-30 16:54:48,530 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/a9205a1bb7fc4c8685bc0a73e3337cc7 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/a9205a1bb7fc4c8685bc0a73e3337cc7 2023-03-30 16:54:48,534 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/a9205a1bb7fc4c8685bc0a73e3337cc7, entries=150, sequenceid=369, filesize=11.9 K 2023-03-30 16:54:48,535 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 2720c59305e818889603edf8ccad1c2c in 1268ms, sequenceid=369, compaction requested=false 2023-03-30 16:54:48,535 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:48,535 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:48,535 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:48,535 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:48,535 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,535 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:48,535 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:48,537 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,537 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:48,537 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,537 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:48,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:48,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:48,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:48,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60c90ec6[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:48,540 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:48,540 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,541 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,541 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:48,542 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:48,542 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:48,542 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,542 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:48,544 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:48,544 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:48,544 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:48,544 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:48,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:48,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:48,547 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:48,547 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:48,547 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:48,547 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:48,547 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:48,549 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,549 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:48,549 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:48,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:48,549 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,551 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,554 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,554 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:48,554 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:48,554 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,554 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,554 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,554 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,554 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:48,554 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:48,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:48,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:48,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:48,554 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:48,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:48,555 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:48,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:48,555 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:48,655 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:48,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:48,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:48,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:48,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:48,662 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:48,662 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:48,662 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:48,662 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:48,662 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,662 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:48,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:48,664 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 16:54:48,665 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:48,665 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:48,665 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:48,665 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:48,665 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:48,665 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:48,665 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:48,665 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:48,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:48,666 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,666 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:48,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:48,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:48,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:48,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:48,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:48,670 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:48,670 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:48,670 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:48,672 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:48,672 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:48,672 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:48,672 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:48,672 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:48,672 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:48,672 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:48,672 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:48,672 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,672 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:48,672 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:48,675 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:48,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,676 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:48,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:48,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47212912[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:48,678 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:48,678 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,679 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,679 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:48,679 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:48,679 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:48,680 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,680 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:48,681 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:48,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:48,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:48,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:48,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:48,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:48,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:48,684 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:48,684 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:48,684 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:48,684 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:48,684 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:48,685 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,685 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:48,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:48,685 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:48,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:48,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:48,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:48,686 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:48,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,687 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,692 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:48,692 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:48,692 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,692 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:48,692 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:48,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:48,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:48,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:48,692 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:48,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:48,692 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,692 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,692 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,692 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:48,693 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:48,692 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:48,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:48,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:48,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195288706, max timestamp=1680195288722Num uniques 859; 2023-03-30 16:54:48,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:48,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195288706, max timestamp=1680195288722Num uniques 859; 2023-03-30 16:54:48,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:48,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195288706, max timestamp=1680195288722Num uniques 859; 2023-03-30 16:54:48,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:48,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:48,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195348776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,793 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:48,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:48,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:48,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:48,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:48,800 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:48,800 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:48,800 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:48,800 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:48,801 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,801 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:48,802 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,802 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:48,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:48,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:48,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:48,803 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,803 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:48,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:48,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:48,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:48,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:48,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:48,805 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:48,805 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:48,805 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:48,806 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:48,806 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:48,806 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:48,806 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:48,806 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:48,806 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:48,806 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:48,806 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:48,806 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,806 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:48,806 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:48,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,808 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:48,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:48,808 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,808 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:48,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:48,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:48,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:48,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:48,810 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:48,811 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3804240a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:48,812 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,812 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:48,812 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:48,812 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:48,812 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,812 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:48,814 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:48,814 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:48,814 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,814 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:48,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:48,818 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:48,818 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:48,818 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:48,818 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:48,818 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:48,819 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,819 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:48,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,819 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:48,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,821 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:48,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,823 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,830 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:48,830 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:48,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:48,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:48,830 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,830 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:48,830 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:48,830 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,830 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,830 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,830 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,831 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:48,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:48,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:48,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:48,831 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:48,831 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:48,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:48,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195348881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,931 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:48,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:48,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:48,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:48,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:48,934 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:48,935 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:48,935 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:48,935 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:48,935 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,935 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:48,937 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:48,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:48,937 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:48,938 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,938 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:48,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:48,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:48,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:48,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:48,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:48,939 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:48,939 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:48,939 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:48,939 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:48,941 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,941 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:48,941 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,941 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:48,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:48,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74f53fc3[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:48,943 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:48,943 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,944 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,944 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:48,946 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,946 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:48,946 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:48,946 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:48,948 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:48,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:48,950 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:48,950 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:48,950 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:48,950 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:48,950 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:48,953 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:48,954 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,955 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,957 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,957 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:48,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,958 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,958 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:48,958 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:48,958 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:48,958 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,958 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:48,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:48,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:48,958 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:48,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:48,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:48,958 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:48,958 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:48,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:48,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:48,959 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:48,959 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:49,059 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:49,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:49,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:49,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:49,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:49,062 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:49,062 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:49,062 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:49,062 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:49,062 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,062 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,063 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:49,063 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:49,064 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,064 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:49,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:49,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:49,065 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:49,065 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:49,065 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:49,065 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:49,067 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,067 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,067 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,067 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:49,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c62b0a5[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:49,068 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:49,068 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,069 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,069 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:49,069 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,069 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:49,069 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:49,069 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:49,075 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:49,075 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:49,075 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,075 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,076 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/80ada46f04f84f739f50f50ffdc822ef 2023-03-30 16:54:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:49,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:49,078 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:49,078 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:49,078 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:49,078 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:49,078 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:49,080 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,080 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,080 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:49,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,081 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,082 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195349083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,087 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:49,087 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:49,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:49,087 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:49,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:49,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:49,087 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:49,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:49,087 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:49,087 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,087 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:49,087 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,088 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,088 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,087 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:49,088 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,188 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:49,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:49,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:49,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:49,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:49,191 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:49,191 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:49,191 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:49,191 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:49,191 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,191 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,192 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:49,192 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:49,193 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,193 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:49,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:49,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:49,194 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:49,194 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:49,194 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:49,194 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:49,194 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:49,194 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:49,194 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:49,194 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:49,194 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:49,194 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:49,194 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:49,194 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,194 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:49,195 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:49,197 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,197 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,197 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,197 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:49,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12c56b82[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:49,199 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:49,199 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,200 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,200 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:49,200 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,200 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:49,200 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:49,200 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:49,201 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,201 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:49,201 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,201 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:49,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:49,204 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:49,204 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:49,204 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:49,204 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:49,204 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:49,205 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,205 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,205 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,206 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,207 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,210 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,210 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,210 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,210 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,210 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:49,210 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:49,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:49,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:49,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:49,210 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,211 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,211 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:49,211 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:49,210 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:49,211 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:49,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:49,311 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:49,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:49,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:49,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:49,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:49,314 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:49,314 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:49,315 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:49,315 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:49,315 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,315 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,323 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,323 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:49,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:49,323 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,323 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:49,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:49,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:49,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:49,325 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:49,325 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:49,325 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:49,325 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:49,325 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:49,325 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:49,325 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:49,325 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:49,325 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,325 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:49,325 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:49,326 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,326 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,326 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,327 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:49,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b7829a4[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:49,328 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:49,328 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,330 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,330 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,330 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:49,330 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:49,330 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:49,330 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:49,331 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:49,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:49,333 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:49,333 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:49,333 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:49,333 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:49,333 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:49,335 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,335 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,335 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:49,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,336 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,337 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,339 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:49,339 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:49,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:49,339 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:49,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:49,339 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:49,339 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,339 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,339 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,339 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,339 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:49,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:49,339 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:49,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:49,339 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,339 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:49,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195349387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,439 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:49,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:49,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:49,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:49,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:49,443 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:49,443 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:49,443 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:49,443 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:49,443 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,443 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,446 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,446 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:49,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:49,446 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,446 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:49,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:49,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:49,447 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:49,447 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:49,447 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:49,447 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:49,447 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:49,447 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:49,447 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:49,447 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:49,447 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:49,447 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:49,448 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:49,448 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,448 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:49,448 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:49,449 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,449 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,449 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,450 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:49,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:49,451 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:49,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56ae042f[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:49,451 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,452 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,452 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:49,452 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:49,452 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:49,452 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,452 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:49,454 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,454 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:49,454 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,454 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:49,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:49,456 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:49,456 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:49,456 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:49,456 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:49,456 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:49,459 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,459 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,459 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:49,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:49,459 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,460 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,462 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:49,462 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:49,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:49,462 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:49,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:49,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:49,462 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,463 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,463 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,463 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,463 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,463 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,463 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,463 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:49,463 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:49,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:49,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:49,487 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/9fed90faffb641518a0cfec9f97f3b42 2023-03-30 16:54:49,563 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:49,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:49,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:49,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:49,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34456 deadline: 1680195349572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:49,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:49,581 DEBUG [Thread-1451] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:54:49,581 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:49,581 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:49,581 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:49,581 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:49,581 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,581 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,583 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:49,583 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:49,583 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,583 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:49,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:49,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:49,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:49,584 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:49,584 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:49,584 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:49,588 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:49,588 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:49,588 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:49,588 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:49,588 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:49,588 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:49,588 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:49,588 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:49,588 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,588 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:49,588 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:49,590 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,590 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,590 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,590 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:49,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f0061c7[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:49,592 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:49,592 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,594 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,594 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:49,594 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:49,594 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:49,594 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,594 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:49,595 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:49,595 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:49,595 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:49,595 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:49,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:49,598 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:49,598 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:49,598 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:49,598 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:49,598 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:49,599 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,599 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,599 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:49,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,600 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,601 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,603 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,603 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,603 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,603 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,603 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,603 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:49,604 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:49,604 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:49,604 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,604 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:49,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:49,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:49,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:49,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:49,604 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:49,604 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:49,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:49,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:49,704 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:49,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:49,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:49,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:49,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:49,707 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:49,707 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:49,707 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:49,707 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:49,708 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,708 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,710 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:49,710 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:49,710 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,710 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:49,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:49,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:49,712 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:49,712 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:49,712 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:49,713 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:49,713 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:49,713 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:49,713 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:49,713 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:49,713 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:49,713 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:49,713 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:49,713 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,713 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:49,713 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:49,719 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,719 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,720 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,720 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:49,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:49,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@262090de[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:49,721 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:49,721 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,722 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,722 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:49,722 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:49,722 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,722 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:49,722 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:49,724 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,725 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,725 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:49,725 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:49,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:49,727 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:49,727 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:49,727 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:49,727 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:49,727 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:49,729 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,729 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:49,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,729 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:49,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:49,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,729 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,731 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:49,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:49,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:49,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:49,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:49,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:49,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:49,734 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:49,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:49,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:49,734 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:49,734 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:49,834 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:49,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:49,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:49,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:49,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:49,838 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:49,838 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:49,838 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:49,838 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:49,839 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,839 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,841 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:49,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:49,841 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,841 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,841 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:49,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:49,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:49,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:49,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:49,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:49,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:49,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:49,844 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:49,844 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:49,844 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:49,844 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:49,844 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:49,844 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:49,844 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:49,844 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:49,844 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,844 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:49,844 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:49,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:49,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34442 deadline: 1680195349842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,847 DEBUG [Thread-1447] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:54:49,848 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,848 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,848 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,848 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:49,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cdfa8b7[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:49,850 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:49,850 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,851 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,851 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:49,851 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:49,851 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:49,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:49,853 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:49,853 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:49,853 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,853 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:49,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34482 deadline: 1680195349850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,855 DEBUG [Thread-1445] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:54:49,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:49,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:49,857 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:49,857 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:49,857 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:49,857 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:49,857 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:49,859 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,859 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,859 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:49,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,859 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:49,861 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:49,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:49,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:49,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,864 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:49,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:49,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:49,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:49,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:49,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:49,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:49,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:49,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,864 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:49,864 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:49,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:49,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34414 deadline: 1680195349862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,866 DEBUG [Thread-1443] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8174 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:54:49,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:49,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195349891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,897 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/83ae7aa098ad4ac480a69dfaf732786f 2023-03-30 16:54:49,901 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/80ada46f04f84f739f50f50ffdc822ef as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/80ada46f04f84f739f50f50ffdc822ef 2023-03-30 16:54:49,911 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/80ada46f04f84f739f50f50ffdc822ef, entries=200, sequenceid=382, filesize=14.3 K 2023-03-30 16:54:49,912 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/9fed90faffb641518a0cfec9f97f3b42 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/9fed90faffb641518a0cfec9f97f3b42 2023-03-30 16:54:49,916 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/9fed90faffb641518a0cfec9f97f3b42, entries=150, sequenceid=382, filesize=11.9 K 2023-03-30 16:54:49,917 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/83ae7aa098ad4ac480a69dfaf732786f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/83ae7aa098ad4ac480a69dfaf732786f 2023-03-30 16:54:49,921 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/83ae7aa098ad4ac480a69dfaf732786f, entries=150, sequenceid=382, filesize=11.9 K 2023-03-30 16:54:49,922 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2720c59305e818889603edf8ccad1c2c in 1258ms, sequenceid=382, compaction requested=true 2023-03-30 16:54:49,922 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:49,922 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:49,922 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:54:49,923 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:49,923 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:54:49,923 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39544 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:54:49,923 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/A is initiating minor compaction (all files) 2023-03-30 16:54:49,924 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/A in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:49,924 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/0757e7cf841e4249ab09d837230af77d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/85946b340f584ac28cbe3e23d06d0bde, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/80ada46f04f84f739f50f50ffdc822ef] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=38.6 K 2023-03-30 16:54:49,923 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:49,924 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 0757e7cf841e4249ab09d837230af77d, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1680195286274 2023-03-30 16:54:49,925 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:54:49,925 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/B is initiating minor compaction (all files) 2023-03-30 16:54:49,925 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 85946b340f584ac28cbe3e23d06d0bde, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1680195286385 2023-03-30 16:54:49,925 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/B in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:49,925 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/4f8a8ac4a5c849f592198486a398e066, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/513d4cedfc074435b31740a80df5866e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/9fed90faffb641518a0cfec9f97f3b42] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=36.2 K 2023-03-30 16:54:49,925 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 80ada46f04f84f739f50f50ffdc822ef, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1680195287537 2023-03-30 16:54:49,925 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 4f8a8ac4a5c849f592198486a398e066, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1680195286274 2023-03-30 16:54:49,926 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 513d4cedfc074435b31740a80df5866e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1680195286385 2023-03-30 16:54:49,926 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 9fed90faffb641518a0cfec9f97f3b42, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1680195287537 2023-03-30 16:54:49,952 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#A#compaction#232 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:49,952 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#B#compaction#231 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:49,965 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:49,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:49,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:49,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:49,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:49,974 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:49,974 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:49,974 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:49,974 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:49,975 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,975 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,976 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:49,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:49,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:49,976 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,977 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:49,977 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:49,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:49,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:49,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:49,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:49,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:49,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:49,992 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:49,992 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:49,992 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:49,992 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:49,993 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-30 16:54:49,993 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:49,993 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:49,993 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:49,993 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:49,993 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:49,993 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:50,017 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/87cb4ea383b04886a89fb54142018b5b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/87cb4ea383b04886a89fb54142018b5b 2023-03-30 16:54:50,017 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/58b269f1a7c9442880799a55a7ab4d03 2023-03-30 16:54:50,023 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/B of 2720c59305e818889603edf8ccad1c2c into 87cb4ea383b04886a89fb54142018b5b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:50,023 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:50,023 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/B, priority=13, startTime=1680195289923; duration=0sec 2023-03-30 16:54:50,023 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:50,023 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:54:50,024 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:54:50,024 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/C is initiating minor compaction (all files) 2023-03-30 16:54:50,024 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/C in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:50,025 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/d912324bc74b479ba8fa9d6bd2e5eb80, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/a9205a1bb7fc4c8685bc0a73e3337cc7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/83ae7aa098ad4ac480a69dfaf732786f] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=36.2 K 2023-03-30 16:54:50,025 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting d912324bc74b479ba8fa9d6bd2e5eb80, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1680195286274 2023-03-30 16:54:50,026 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting a9205a1bb7fc4c8685bc0a73e3337cc7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=369, earliestPutTs=1680195286385 2023-03-30 16:54:50,026 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 83ae7aa098ad4ac480a69dfaf732786f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1680195287537 2023-03-30 16:54:50,033 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#C#compaction#235 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:50,419 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/d4ba6ca8b0db4a83938a81538f87d19d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/d4ba6ca8b0db4a83938a81538f87d19d 2023-03-30 16:54:50,423 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/A of 2720c59305e818889603edf8ccad1c2c into d4ba6ca8b0db4a83938a81538f87d19d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:50,423 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:50,423 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/A, priority=13, startTime=1680195289922; duration=0sec 2023-03-30 16:54:50,423 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:50,439 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/946467c91efc4940b73d2ac3ae49eeed 2023-03-30 16:54:50,454 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/0746da17b0844507a331b3ff81e2ba2a 2023-03-30 16:54:50,459 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/58b269f1a7c9442880799a55a7ab4d03 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/58b269f1a7c9442880799a55a7ab4d03 2023-03-30 16:54:50,463 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/58b269f1a7c9442880799a55a7ab4d03, entries=150, sequenceid=418, filesize=11.9 K 2023-03-30 16:54:50,465 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/946467c91efc4940b73d2ac3ae49eeed as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/946467c91efc4940b73d2ac3ae49eeed 2023-03-30 16:54:50,469 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/946467c91efc4940b73d2ac3ae49eeed, entries=150, sequenceid=418, filesize=11.9 K 2023-03-30 16:54:50,470 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/0746da17b0844507a331b3ff81e2ba2a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/0746da17b0844507a331b3ff81e2ba2a 2023-03-30 16:54:50,473 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/0746da17b0844507a331b3ff81e2ba2a, entries=150, sequenceid=418, filesize=11.9 K 2023-03-30 16:54:50,474 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for 2720c59305e818889603edf8ccad1c2c in 481ms, sequenceid=418, compaction requested=false 2023-03-30 16:54:50,474 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:50,474 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:50,474 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:50,474 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:50,474 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,474 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:50,474 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:50,476 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,476 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:50,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:50,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:50,478 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,478 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:50,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:50,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4af6467f[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:50,479 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:50,479 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/e79458c7b7254fa79f9f0e996bd3bbf6 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/e79458c7b7254fa79f9f0e996bd3bbf6 2023-03-30 16:54:50,480 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,481 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,481 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:50,481 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:50,481 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:50,482 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,482 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:50,483 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,483 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:50,483 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:50,484 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,486 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/C of 2720c59305e818889603edf8ccad1c2c into e79458c7b7254fa79f9f0e996bd3bbf6(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:50,486 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:50,486 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/C, priority=13, startTime=1680195289923; duration=0sec 2023-03-30 16:54:50,486 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:50,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:50,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:50,487 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:50,487 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:50,487 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:50,487 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:50,487 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:50,489 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,489 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,489 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:50,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:50,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:50,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:50,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,490 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,492 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,494 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,494 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:50,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:50,494 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:50,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:50,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:50,494 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:50,495 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:50,495 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:50,494 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:50,495 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,494 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:50,495 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:50,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:50,495 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,495 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:50,595 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:50,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:50,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:50,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:50,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:50,598 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:50,598 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:50,598 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:50,598 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:50,599 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,599 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:50,600 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:50,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:50,600 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:50,600 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,601 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:50,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:50,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:50,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:50,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:50,602 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:50,602 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:50,602 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:50,602 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:50,602 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:50,602 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:50,602 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:50,602 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:50,602 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:50,602 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:50,603 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:50,603 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,603 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:50,603 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:50,605 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,605 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:50,605 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,605 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:50,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:50,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@566418bf[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:50,607 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:50,607 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,608 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,608 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:50,608 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:50,608 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:50,608 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,608 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:50,612 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:50,612 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:50,612 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,612 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:50,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:50,615 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:50,615 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:50,615 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:50,615 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:50,615 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:50,618 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,618 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,618 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:50,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:50,619 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:50,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,620 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:50,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:50,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:50,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:50,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:50,622 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:50,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,623 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:50,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:50,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:50,622 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:50,623 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:50,623 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:50,623 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,623 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:50,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:50,723 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:50,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:50,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:50,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:50,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:50,726 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:50,726 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:50,726 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:50,726 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:50,727 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,727 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:50,728 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,728 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:50,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:50,728 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,728 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:50,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:50,729 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:50,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:50,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:50,730 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:50,730 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:50,730 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:50,730 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:50,730 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:50,730 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:50,730 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:50,730 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:50,730 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:50,730 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:50,730 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:50,730 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,730 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:50,730 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:50,732 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,732 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:50,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:50,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74e0c4cd[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:50,734 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:50,734 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,738 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,738 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:50,738 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:50,738 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:50,738 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,739 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:50,740 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:50,740 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:50,740 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,740 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:50,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:50,742 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:50,743 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:50,743 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:50,743 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:50,743 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:50,744 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,744 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:50,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,744 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:50,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,745 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:50,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:50,746 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:50,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:50,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:50,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:50,748 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:50,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:50,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:50,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:50,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:50,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:50,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,749 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,749 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:50,749 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:50,749 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:50,849 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:50,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:50,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:50,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:50,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:50,852 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:50,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:50,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:50,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:50,853 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,853 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:50,855 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:50,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:50,855 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:50,856 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,856 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:50,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:50,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:50,857 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:50,857 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:50,857 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:50,857 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:50,857 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:50,858 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:50,858 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:50,858 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:50,858 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:50,858 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:50,858 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:50,858 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:50,858 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:50,858 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,858 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:50,858 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:50,860 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,860 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:50,860 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:50,860 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:50,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:50,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:50,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@656b50f6[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:50,862 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:50,862 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,863 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,863 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:50,863 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:50,863 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:50,863 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,863 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:50,865 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,865 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:50,865 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:50,865 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:50,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:50,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:50,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:50,867 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:50,868 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:50,868 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:50,868 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:50,868 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:50,871 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,871 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:50,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,871 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:50,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:50,872 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:50,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,873 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:50,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:50,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,876 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:50,876 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:50,876 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:50,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:50,876 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:50,876 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:50,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:50,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:50,876 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:50,876 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:50,876 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:50,876 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:50,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:50,876 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,877 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:50,877 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:50,912 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:54:50,912 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:50,912 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:50,912 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:50,912 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:50,912 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:50,912 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:50,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195290952, max timestamp=1680195290964Num uniques 859; 2023-03-30 16:54:50,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195290952, max timestamp=1680195290964Num uniques 859; 2023-03-30 16:54:50,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195290952, max timestamp=1680195290964Num uniques 859; 2023-03-30 16:54:50,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:50,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:50,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:50,977 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:50,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:50,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:50,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:50,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:50,988 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:50,988 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:50,988 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:50,988 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:50,988 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,988 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:50,990 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:50,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:50,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:50,990 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:50,990 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:50,990 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:50,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:50,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:50,991 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:50,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:50,992 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:50,992 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:50,992 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:51,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:51,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:51,012 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:51,012 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:51,012 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:51,012 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:51,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:51,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:51,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:51,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:51,014 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,015 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,015 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:51,020 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:51,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:51,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d56379a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:51,023 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:51,023 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,025 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,025 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:51,025 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:51,025 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:51,026 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,026 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:51,027 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,027 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:51,027 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:51,027 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:51,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:51,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:51,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:51,029 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:51,029 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:51,029 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:51,029 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:51,029 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:51,034 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,034 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,034 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:51,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:51,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:51,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:51,035 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:51,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,038 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,043 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:51,043 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:51,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:51,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:51,043 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,043 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,043 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,043 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,043 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,043 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:51,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:51,043 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:51,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:51,043 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,044 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:51,044 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:51,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:51,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195351083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,144 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:51,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:51,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:51,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:51,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:51,147 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:51,147 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:51,147 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:51,147 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:51,148 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,148 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,149 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:51,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:51,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:51,149 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,149 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,150 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:51,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:51,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:51,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:51,150 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:51,150 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:51,150 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:51,150 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:51,151 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:51,151 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:51,151 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:51,151 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:51,151 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:51,151 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:51,151 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:51,151 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,151 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:51,151 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:51,152 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,152 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:51,153 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,153 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:51,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:51,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b01756f[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:51,155 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:51,155 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,156 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,156 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:51,156 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:51,156 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:51,156 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,156 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:51,158 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,158 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:51,158 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:51,158 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:51,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:51,161 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:51,161 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:51,161 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:51,161 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:51,161 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:51,163 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,163 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,163 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:51,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:51,164 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,165 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,167 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:51,167 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:51,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:51,168 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:51,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:51,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:51,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:51,168 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,168 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,168 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,168 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,168 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,168 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:51,168 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:51,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:51,168 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,168 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:51,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195351189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,268 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:51,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:51,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:51,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:51,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:51,271 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:51,271 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:51,271 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:51,271 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:51,272 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,272 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,274 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,274 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:51,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:51,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:51,274 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,274 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:51,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:51,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:51,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:51,276 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:51,276 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:51,276 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:51,277 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:51,277 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:51,277 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:51,277 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:51,277 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:51,277 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:51,277 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:51,277 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:51,278 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,278 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:51,278 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:51,280 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:51,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:51,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:51,281 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:51,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1935026e[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:51,281 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,282 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,283 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:51,283 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:51,283 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:51,283 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,283 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:51,284 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,284 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:51,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:51,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:51,284 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:51,284 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:51,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:51,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:51,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:51,286 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:51,286 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:51,286 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:51,286 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:51,286 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:51,288 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,288 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:51,288 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:51,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,289 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:51,290 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,292 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,292 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,292 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:51,292 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:51,292 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:51,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:51,292 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:51,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:51,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:51,293 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,293 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,293 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:51,293 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,293 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:51,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:51,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:51,293 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,320 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/3651f93c436f44088d019d31918ba9a4 2023-03-30 16:54:51,336 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/e7368180d4a74b6bbe9ee78b29f1c693 2023-03-30 16:54:51,393 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:51,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:51,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:51,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:51,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195351394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:51,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:51,396 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:51,396 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:51,397 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:51,397 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:51,397 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,397 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,398 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,398 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:51,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:51,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:51,399 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,399 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:51,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:51,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:51,400 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:51,400 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:51,400 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:51,400 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:51,400 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:51,400 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:51,400 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:51,400 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:51,400 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:51,400 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:51,401 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:51,401 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,401 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:51,401 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:51,403 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,403 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:51,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:51,403 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,403 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:51,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:51,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:51,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:51,404 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:51,404 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26660d5e[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:51,406 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,406 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,406 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:51,406 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:51,406 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:51,406 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:51,408 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,408 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:51,408 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:51,408 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:51,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:51,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:51,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:51,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:51,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:51,411 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:51,411 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:51,411 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:51,411 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:51,411 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:51,413 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,413 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,413 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:51,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:51,414 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:51,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,415 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:51,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:51,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:51,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:51,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:51,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:51,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:51,419 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:51,419 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:51,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:51,419 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:51,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:51,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:51,520 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:51,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:51,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:51,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:51,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:51,523 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:51,523 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:51,523 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:51,523 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:51,523 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,523 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,525 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,525 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:51,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:51,526 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,526 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:51,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:51,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:51,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:51,527 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:51,527 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:51,527 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:51,527 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:51,527 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:51,527 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:51,527 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:51,527 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:51,527 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:51,527 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:51,527 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:51,528 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,528 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:51,528 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:51,529 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,529 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:51,529 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,529 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:51,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:51,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a2014ef[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:51,531 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:51,531 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,533 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,533 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:51,533 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:51,533 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:51,533 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,533 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:51,535 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:51,535 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,535 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:51,535 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:51,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:51,537 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:51,537 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:51,537 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:51,537 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:51,537 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:51,540 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,540 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,540 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:51,541 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:51,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,542 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,544 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,544 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,544 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,544 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:51,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:51,544 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:51,544 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,544 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,544 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,544 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:51,544 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:51,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:51,544 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:51,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:51,544 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:51,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:51,645 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:51,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:51,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:51,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:51,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:51,648 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:51,648 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:51,648 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:51,648 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:51,648 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,648 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,650 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:51,650 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:51,650 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,650 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:51,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:51,651 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:51,651 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:51,651 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:51,651 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:51,651 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:51,651 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:51,651 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:51,651 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:51,651 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:51,651 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:51,652 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:51,652 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,652 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:51,652 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:51,653 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,653 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:51,653 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,653 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:51,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:51,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a6ab2c9[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:51,655 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:51,655 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,657 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,657 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:51,657 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:51,657 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:51,658 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,658 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:51,659 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:51,659 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:51,659 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:51,660 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:51,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:51,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:51,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:51,662 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:51,662 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:51,662 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:51,662 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:51,663 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:51,664 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,664 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:51,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,664 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:51,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,666 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,667 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,670 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:51,670 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,670 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:51,670 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,670 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:51,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:51,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:51,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:51,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:51,670 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,670 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,670 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,670 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:51,671 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,671 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:51,671 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:51,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:51,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:51,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195351699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,745 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/149f2accf2a1440d93186e1d0ee7dfe3 2023-03-30 16:54:51,752 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/3651f93c436f44088d019d31918ba9a4 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/3651f93c436f44088d019d31918ba9a4 2023-03-30 16:54:51,755 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/3651f93c436f44088d019d31918ba9a4, entries=200, sequenceid=434, filesize=14.3 K 2023-03-30 16:54:51,756 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/e7368180d4a74b6bbe9ee78b29f1c693 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/e7368180d4a74b6bbe9ee78b29f1c693 2023-03-30 16:54:51,762 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/e7368180d4a74b6bbe9ee78b29f1c693, entries=150, sequenceid=434, filesize=11.9 K 2023-03-30 16:54:51,763 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/149f2accf2a1440d93186e1d0ee7dfe3 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/149f2accf2a1440d93186e1d0ee7dfe3 2023-03-30 16:54:51,769 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/149f2accf2a1440d93186e1d0ee7dfe3, entries=150, sequenceid=434, filesize=11.9 K 2023-03-30 16:54:51,770 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2720c59305e818889603edf8ccad1c2c in 858ms, sequenceid=434, compaction requested=true 2023-03-30 16:54:51,770 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:51,770 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:51,770 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:54:51,770 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:54:51,770 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:51,771 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:51,771 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:51,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:51,771 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39646 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:54:51,772 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/A is initiating minor compaction (all files) 2023-03-30 16:54:51,772 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/A in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:51,772 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/d4ba6ca8b0db4a83938a81538f87d19d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/58b269f1a7c9442880799a55a7ab4d03, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/3651f93c436f44088d019d31918ba9a4] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=38.7 K 2023-03-30 16:54:51,772 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:54:51,772 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/B is initiating minor compaction (all files) 2023-03-30 16:54:51,773 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/B in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:51,773 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/87cb4ea383b04886a89fb54142018b5b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/946467c91efc4940b73d2ac3ae49eeed, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/e7368180d4a74b6bbe9ee78b29f1c693] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=36.3 K 2023-03-30 16:54:51,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:51,773 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting d4ba6ca8b0db4a83938a81538f87d19d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1680195287537 2023-03-30 16:54:51,773 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 87cb4ea383b04886a89fb54142018b5b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1680195287537 2023-03-30 16:54:51,773 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 58b269f1a7c9442880799a55a7ab4d03, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1680195288762 2023-03-30 16:54:51,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:51,774 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 3651f93c436f44088d019d31918ba9a4, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1680195290902 2023-03-30 16:54:51,774 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 946467c91efc4940b73d2ac3ae49eeed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1680195288762 2023-03-30 16:54:51,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:51,774 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:51,774 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:51,774 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:51,774 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:51,775 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,775 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,775 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting e7368180d4a74b6bbe9ee78b29f1c693, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1680195290907 2023-03-30 16:54:51,777 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:51,777 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,778 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:51,778 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:51,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:51,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:51,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:51,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:51,779 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:51,779 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:51,779 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:51,779 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:51,779 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:51,780 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:51,780 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:51,780 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-30 16:54:51,780 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:51,780 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:51,780 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:51,781 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:51,781 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:51,781 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:51,782 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#B#compaction#241 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:51,782 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#A#compaction#240 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:51,800 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/dbee5f3cdf1e424980b8371a6ca3bfe6 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/dbee5f3cdf1e424980b8371a6ca3bfe6 2023-03-30 16:54:51,805 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/A of 2720c59305e818889603edf8ccad1c2c into dbee5f3cdf1e424980b8371a6ca3bfe6(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:51,805 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:51,805 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/A, priority=13, startTime=1680195291770; duration=0sec 2023-03-30 16:54:51,805 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:51,805 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:54:51,806 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:54:51,806 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/C is initiating minor compaction (all files) 2023-03-30 16:54:51,806 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/C in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:51,806 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/e79458c7b7254fa79f9f0e996bd3bbf6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/0746da17b0844507a331b3ff81e2ba2a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/149f2accf2a1440d93186e1d0ee7dfe3] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=36.3 K 2023-03-30 16:54:51,806 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting e79458c7b7254fa79f9f0e996bd3bbf6, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1680195287537 2023-03-30 16:54:51,807 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 0746da17b0844507a331b3ff81e2ba2a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1680195288762 2023-03-30 16:54:51,808 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 149f2accf2a1440d93186e1d0ee7dfe3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1680195290907 2023-03-30 16:54:51,824 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#C#compaction#243 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:51,833 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/d4a88e39c4ce46f996af350011242962 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/d4a88e39c4ce46f996af350011242962 2023-03-30 16:54:51,841 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/C of 2720c59305e818889603edf8ccad1c2c into d4a88e39c4ce46f996af350011242962(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:51,841 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:51,841 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/C, priority=13, startTime=1680195291771; duration=0sec 2023-03-30 16:54:51,841 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:52,202 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/33fcda4dd5bb4d3b92ebe5f16c3e0db5 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/33fcda4dd5bb4d3b92ebe5f16c3e0db5 2023-03-30 16:54:52,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:52,207 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:52,210 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/B of 2720c59305e818889603edf8ccad1c2c into 33fcda4dd5bb4d3b92ebe5f16c3e0db5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:52,210 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:52,210 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/B, priority=13, startTime=1680195291770; duration=0sec 2023-03-30 16:54:52,210 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:52,218 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/9db401505d874233995640121d24423e 2023-03-30 16:54:52,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:52,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195352224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:52,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:52,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195352326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:52,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:52,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195352530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:52,633 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/09c3532d8f804d588ad7ad37405aa361 2023-03-30 16:54:52,645 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/8ebfd50998d1486d851ddad2458334c0 2023-03-30 16:54:52,649 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/9db401505d874233995640121d24423e as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/9db401505d874233995640121d24423e 2023-03-30 16:54:52,653 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/9db401505d874233995640121d24423e, entries=150, sequenceid=471, filesize=11.9 K 2023-03-30 16:54:52,653 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/09c3532d8f804d588ad7ad37405aa361 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/09c3532d8f804d588ad7ad37405aa361 2023-03-30 16:54:52,657 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/09c3532d8f804d588ad7ad37405aa361, entries=150, sequenceid=471, filesize=11.9 K 2023-03-30 16:54:52,657 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/8ebfd50998d1486d851ddad2458334c0 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/8ebfd50998d1486d851ddad2458334c0 2023-03-30 16:54:52,661 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/8ebfd50998d1486d851ddad2458334c0, entries=150, sequenceid=471, filesize=11.9 K 2023-03-30 16:54:52,661 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 2720c59305e818889603edf8ccad1c2c in 881ms, sequenceid=471, compaction requested=false 2023-03-30 16:54:52,662 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:52,662 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:52,662 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:52,662 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:52,662 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:52,662 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:52,662 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:52,663 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:52,663 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:52,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:52,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:52,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:52,664 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:52,664 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:52,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:52,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:52,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:52,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:52,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:52,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:52,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bfdfb98[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:52,665 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:52,665 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:52,666 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:52,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:52,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:52,666 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:52,666 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:52,666 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:52,666 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:52,667 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:52,668 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:52,668 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:52,668 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:52,668 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:52,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:52,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:52,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:52,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:52,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:52,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:52,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:52,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:52,670 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:52,670 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:52,670 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:52,670 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:52,671 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:52,678 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:52,678 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:52,678 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:52,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:52,679 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:52,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:52,680 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:52,682 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:52,682 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:52,682 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:52,682 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:52,682 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:52,683 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:52,683 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:52,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:52,683 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:52,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:52,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:52,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:52,683 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:52,683 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:52,683 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:52,683 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:52,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:52,783 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:52,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:52,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:52,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:52,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:52,786 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:52,786 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:52,787 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:52,787 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:52,787 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:52,787 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:52,791 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:52,791 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:52,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:52,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:52,791 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:52,791 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:52,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:52,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:52,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:52,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:52,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:52,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:52,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:52,792 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:52,792 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:52,792 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:52,792 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:52,792 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:52,792 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:52,792 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:52,793 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:54:52,793 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:52,793 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:52,793 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:52,793 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:52,793 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:52,793 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:52,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:52,834 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:52,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195292863, max timestamp=1680195292871Num uniques 859; 2023-03-30 16:54:52,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195292863, max timestamp=1680195292871Num uniques 859; 2023-03-30 16:54:52,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:52,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195292863, max timestamp=1680195292871Num uniques 859; 2023-03-30 16:54:52,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:52,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:52,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:52,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195352917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:53,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:53,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195353021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:53,201 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/ece3c8fed2d54804a4e3a6d545ebfb2a 2023-03-30 16:54:53,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:53,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195353223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:53,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:53,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195353525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:53,611 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/0e696543633b4b71bf0a6ae427b35ff0 2023-03-30 16:54:54,022 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/b0b6eceff6414c4a858c8b191f1f000c 2023-03-30 16:54:54,025 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/ece3c8fed2d54804a4e3a6d545ebfb2a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/ece3c8fed2d54804a4e3a6d545ebfb2a 2023-03-30 16:54:54,029 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/ece3c8fed2d54804a4e3a6d545ebfb2a, entries=150, sequenceid=484, filesize=11.9 K 2023-03-30 16:54:54,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:54,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195354028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,030 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/0e696543633b4b71bf0a6ae427b35ff0 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/0e696543633b4b71bf0a6ae427b35ff0 2023-03-30 16:54:54,033 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/0e696543633b4b71bf0a6ae427b35ff0, entries=150, sequenceid=484, filesize=11.9 K 2023-03-30 16:54:54,033 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/b0b6eceff6414c4a858c8b191f1f000c as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/b0b6eceff6414c4a858c8b191f1f000c 2023-03-30 16:54:54,037 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/b0b6eceff6414c4a858c8b191f1f000c, entries=150, sequenceid=484, filesize=11.9 K 2023-03-30 16:54:54,037 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2720c59305e818889603edf8ccad1c2c in 1244ms, sequenceid=484, compaction requested=true 2023-03-30 16:54:54,037 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:54,038 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:54,039 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,039 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,040 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,040 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:54,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d670144[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:54,041 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:54,041 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,042 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,042 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:54,042 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:54,042 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:54,043 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,043 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:54,045 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:54,045 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:54,045 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,045 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:54,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:54,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:54,047 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:54,047 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:54,047 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:54,047 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:54,047 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:54,049 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,049 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,049 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:54,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,049 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,051 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,052 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,052 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,052 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,052 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,053 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,052 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:54,053 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:54,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:54,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:54,053 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:54,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:54,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:54,053 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,053 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:54,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:54,053 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:54,053 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:54,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:54,153 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:54,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:54,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:54,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:54,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:54,164 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:54,164 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:54,164 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:54,164 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:54,165 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,165 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,166 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,166 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:54,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:54,167 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,167 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:54,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:54,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:54,168 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:54,168 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:54,168 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:54,168 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:54,168 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:54,168 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:54,168 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:54,168 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-30 16:54:54,169 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:54,169 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:54,169 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:54,169 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:54,169 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:54,169 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:54,186 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/c48dc92e18fc4222b9775aebd9415e9b 2023-03-30 16:54:54,202 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/522148798a9a4a6bac587e3cef190350 2023-03-30 16:54:54,213 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/9b679f2a311b432ba1243bda12e1a03f 2023-03-30 16:54:54,217 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/c48dc92e18fc4222b9775aebd9415e9b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/c48dc92e18fc4222b9775aebd9415e9b 2023-03-30 16:54:54,220 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/c48dc92e18fc4222b9775aebd9415e9b, entries=150, sequenceid=521, filesize=11.9 K 2023-03-30 16:54:54,221 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/522148798a9a4a6bac587e3cef190350 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/522148798a9a4a6bac587e3cef190350 2023-03-30 16:54:54,226 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/522148798a9a4a6bac587e3cef190350, entries=150, sequenceid=521, filesize=11.9 K 2023-03-30 16:54:54,226 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/9b679f2a311b432ba1243bda12e1a03f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/9b679f2a311b432ba1243bda12e1a03f 2023-03-30 16:54:54,230 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/9b679f2a311b432ba1243bda12e1a03f, entries=150, sequenceid=521, filesize=11.9 K 2023-03-30 16:54:54,231 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 2720c59305e818889603edf8ccad1c2c in 63ms, sequenceid=521, compaction requested=true 2023-03-30 16:54:54,231 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:54,231 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:54,231 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:54,231 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:54,231 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,231 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:54,231 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:54,233 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,233 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,233 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,233 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:54,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:54,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47eaf36b[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:54,235 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:54,235 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,236 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,236 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:54,236 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:54,236 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:54,236 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,236 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:54,238 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:54,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:54,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:54,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:54,241 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:54,241 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:54,241 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:54,241 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:54,241 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:54,243 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,243 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,243 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:54,244 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,245 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,247 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:54,247 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:54,247 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:54,247 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:54,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:54,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:54,247 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,248 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,248 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,248 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,248 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:54,248 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:54,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:54,248 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,248 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,348 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:54,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:54,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:54,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:54,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:54,352 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:54,352 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:54,352 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:54,352 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:54,352 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,352 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,354 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:54,354 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:54,354 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,354 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:54,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:54,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:54,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:54,355 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:54,355 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:54,355 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:54,355 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:54,356 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:54,356 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:54,356 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:54,356 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:54,356 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:54,356 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:54,356 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:54,356 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,356 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:54,356 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:54,357 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,358 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,358 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,358 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:54,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78a9eda5[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:54,359 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:54,359 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,362 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,362 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:54,362 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:54,362 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:54,362 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,362 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:54,363 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,363 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:54,363 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:54,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,363 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:54,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:54,366 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:54,366 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:54,366 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:54,366 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:54,366 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:54,367 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,367 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,367 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,368 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,370 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,372 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,372 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,372 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,372 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,372 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:54,372 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:54,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:54,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:54,372 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:54,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:54,372 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:54,373 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:54,372 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,373 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,373 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:54,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:54,473 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:54,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:54,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:54,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:54,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:54,476 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:54,476 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:54,476 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:54,476 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:54,476 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,477 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,478 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:54,478 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:54,479 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,479 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:54,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:54,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:54,480 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:54,480 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:54,480 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:54,480 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:54,480 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:54,485 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:54,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c3cf813[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:54,486 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:54,486 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,487 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,487 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:54,487 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:54,487 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:54,487 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,487 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:54,489 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,489 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,489 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:54,489 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:54,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:54,491 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:54,491 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:54,491 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:54,491 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:54,491 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:54,494 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,494 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:54,494 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,494 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,495 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,498 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:54,498 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:54,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:54,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:54,500 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,500 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,500 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,500 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:54,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:54,500 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,500 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,500 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,500 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:54,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:54,500 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:54,500 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:54,600 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:54,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:54,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:54,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:54,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:54,604 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:54,604 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:54,604 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:54,604 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:54,604 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,604 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,605 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,605 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:54,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:54,606 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,606 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,607 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:54,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:54,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:54,607 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:54,608 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:54,608 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:54,608 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:54,608 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:54,608 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:54,608 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:54,608 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:54,608 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:54,608 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:54,608 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:54,608 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,608 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:54,608 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:54,609 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,610 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,610 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,610 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:54,611 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:54,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39a4adf1[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:54,611 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,615 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:54,615 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,615 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:54,616 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:54,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:54,618 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:54,619 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:54,619 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:54,619 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:54,619 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:54,620 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,620 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,620 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,621 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,622 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,624 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,624 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,624 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,624 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,624 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:54,624 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,624 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:54,624 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:54,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:54,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:54,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:54,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:54,624 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,625 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:54,625 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:54,625 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:54,725 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:54,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:54,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:54,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:54,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:54,728 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:54,728 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:54,728 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:54,729 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:54,729 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,729 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,731 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,731 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:54,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:54,732 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,732 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:54,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:54,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:54,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:54,733 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:54,733 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:54,733 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:54,734 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:54,734 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:54,734 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:54,734 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:54,734 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:54,734 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:54,734 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:54,734 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:54,734 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,734 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:54,734 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:54,735 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,735 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,736 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,736 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:54,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29457005[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:54,737 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:54,737 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,738 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,738 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:54,738 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,738 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:54,738 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:54,738 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:54,739 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:54,739 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,739 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,740 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:54,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:54,742 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:54,742 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:54,742 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:54,742 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:54,742 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:54,744 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,744 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,744 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,745 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,746 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:54,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:54,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:54,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:54,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:54,748 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:54,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:54,748 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:54,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,749 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:54,749 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:54,749 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,749 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,749 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,749 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,849 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:54,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:54,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:54,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:54,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:54,852 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:54,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:54,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:54,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:54,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,854 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:54,854 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:54,854 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,854 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:54,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:54,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:54,855 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:54,855 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:54,855 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:54,856 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:54,856 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:54,856 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:54,856 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:54,856 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:54,856 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:54,856 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:54,856 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:54,856 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,856 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:54,856 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:54,858 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,858 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,858 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,858 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:54,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb6644d[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:54,864 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:54,865 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,867 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,867 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:54,867 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:54,867 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,867 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:54,867 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:54,871 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,871 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,871 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:54,871 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:54,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:54,873 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:54,873 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:54,873 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:54,873 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:54,873 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:54,875 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,875 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,875 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,875 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,877 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,879 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,879 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,879 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:54,879 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:54,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:54,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:54,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:54,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:54,879 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:54,879 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:54,879 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,879 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,879 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,879 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:54,879 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,879 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:54,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:54,980 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:54,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:54,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:54,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:54,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:54,983 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:54,983 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:54,983 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:54,983 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:54,983 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,983 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,986 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,986 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:54,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:54,986 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,986 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:54,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:54,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:54,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:54,988 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:54,988 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:54,988 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:54,988 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:54,988 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:54,988 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:54,988 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:54,988 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:54,989 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:54,989 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:54,989 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:54,989 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,989 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:54,989 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:54,990 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,990 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:54,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3561ebe8[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:54,993 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:54,993 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,995 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:54,996 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,996 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:54,997 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:54,997 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:54,997 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,997 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:55,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:55,000 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:55,000 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:55,000 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:55,000 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:55,000 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:55,002 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,002 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:55,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,002 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,003 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,004 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,007 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,007 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,007 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,007 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,007 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,007 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,007 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,007 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:55,007 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:55,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:55,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:55,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:55,007 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:55,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:55,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:55,008 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:55,008 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:55,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:55,048 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:54:55,048 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:55,048 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:55,048 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:55,048 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:55,048 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:55,048 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:55,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195295084, max timestamp=1680195295101Num uniques 859; 2023-03-30 16:54:55,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:55,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195295084, max timestamp=1680195295101Num uniques 859; 2023-03-30 16:54:55,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195295084, max timestamp=1680195295101Num uniques 859; 2023-03-30 16:54:55,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:55,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:55,108 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:55,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:55,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:55,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:55,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:55,111 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:55,111 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:55,111 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:55,111 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:55,111 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,112 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,113 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:55,113 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:55,113 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,113 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:55,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:55,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:55,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:55,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:55,114 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:55,114 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:55,114 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:55,114 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:55,114 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:55,114 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:55,114 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:55,115 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:55,115 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:55,115 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:55,115 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:55,115 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,115 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:55,115 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:55,116 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,116 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,116 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,116 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:55,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29b61390[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:55,118 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:55,118 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,124 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,124 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:55,124 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,125 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:55,125 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:55,125 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:55,126 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:55,126 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:55,126 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,126 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:55,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:55,128 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:55,128 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:55,128 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:55,128 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:55,128 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:55,130 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,130 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,130 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:55,131 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,132 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,134 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:55,134 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,134 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:55,134 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:55,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:55,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:55,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:55,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:55,134 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,135 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,135 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,135 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:55,145 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,135 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:55,145 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,145 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:55,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:55,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195355179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,245 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:55,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:55,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:55,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:55,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:55,264 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:55,264 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:55,264 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:55,264 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:55,265 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,265 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,266 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:55,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:55,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:55,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,268 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:55,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:55,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:55,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:55,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:55,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:55,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:55,280 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:55,281 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:55,281 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:55,281 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:55,281 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:55,281 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:55,281 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,281 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:55,281 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:55,283 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,283 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,284 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,284 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:55,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bf19bfb[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:55,285 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:55,285 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:55,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195355283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,287 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,287 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:55,287 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:55,287 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:55,288 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,288 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:55,289 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:55,289 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:55,289 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:55,289 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:55,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:55,292 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:55,292 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:55,292 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:55,292 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:55,292 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:55,300 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,300 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,300 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:55,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:55,300 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,302 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,304 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:55,304 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:55,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:55,304 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:55,304 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,304 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,305 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,305 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,305 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,305 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:55,305 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:55,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:55,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:55,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:55,305 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:55,305 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:55,405 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:55,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:55,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:55,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:55,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:55,413 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:55,413 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:55,413 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:55,413 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:55,413 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,413 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,414 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:55,414 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,415 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,415 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:55,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:55,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:55,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:55,424 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:55,424 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:55,424 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:55,425 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:55,425 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:55,425 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:55,425 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:55,426 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:55,428 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,428 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,428 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,428 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:55,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64806c4[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:55,430 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:55,430 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,431 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,431 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:55,431 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:55,431 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:55,431 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,431 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:55,432 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,432 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:55,432 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,432 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:55,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:55,435 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:55,435 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:55,435 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:55,435 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:55,435 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:55,437 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,437 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:55,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,437 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,438 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,439 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,442 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,442 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,442 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,442 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,442 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,443 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,443 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,443 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:55,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:55,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:55,443 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:55,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:55,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:55,443 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:55,443 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:55,443 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:55,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:55,456 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/d7d3b15f87884e8da550a2eac050b8f9 2023-03-30 16:54:55,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:55,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195355487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,543 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:55,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:55,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:55,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:55,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:55,546 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:55,546 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:55,546 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:55,547 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:55,547 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,547 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,548 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,548 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:55,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:55,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:55,548 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,548 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:55,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:55,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:55,550 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:55,550 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:55,550 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:55,550 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:55,550 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:55,552 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,552 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,552 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,552 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:55,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ccc934e[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:55,553 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:55,553 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,554 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:55,555 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,555 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:55,557 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,557 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,557 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:55,557 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:55,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:55,559 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:55,559 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:55,559 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:55,559 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:55,559 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:55,560 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,560 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,560 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:55,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,561 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,564 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,567 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,567 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,567 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,567 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:55,567 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,567 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:55,567 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:55,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:55,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:55,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:55,567 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,567 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:55,568 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:55,568 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:55,568 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:55,668 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:55,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:55,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:55,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:55,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:55,671 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:55,671 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:55,671 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:55,671 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:55,672 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,672 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,673 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:55,673 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:55,674 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,674 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:55,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:55,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:55,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:55,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:55,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:55,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:55,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:55,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:55,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:55,675 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:55,675 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:55,675 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:55,675 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:55,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:55,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:55,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:55,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:55,678 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,678 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,679 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,679 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:55,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25ff3dd4[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:55,680 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:55,680 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,681 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,681 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:55,682 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:55,683 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,683 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:55,683 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,683 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:55,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:55,685 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:55,685 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:55,685 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:55,685 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:55,685 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:55,687 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,687 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:55,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,687 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,688 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,689 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,695 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:55,695 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:55,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:55,695 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:55,695 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:55,695 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:55,695 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:55,696 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:55,695 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,696 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,696 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:55,696 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,696 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:55,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195355790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,796 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:55,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:55,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:55,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:55,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:55,799 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:55,799 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:55,799 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:55,799 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:55,799 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,799 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,800 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,800 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:55,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:55,801 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,801 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:55,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:55,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:55,802 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:55,802 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:55,802 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:55,802 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:55,802 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:55,802 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:55,802 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:55,802 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:55,802 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:55,802 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:55,802 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:55,802 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,802 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:55,802 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:55,804 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,804 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,804 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,804 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:55,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:55,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27644d3[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:55,805 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:55,805 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,810 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,810 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:55,810 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:55,810 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,810 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:55,810 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:55,811 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:55,812 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:55,812 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,812 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:55,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:55,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:55,814 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:55,814 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:55,814 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:55,814 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:55,814 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:55,815 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,815 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,815 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,816 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,817 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,819 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,819 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:55,819 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:55,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:55,819 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,820 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:55,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:55,820 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,820 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:55,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:55,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:55,820 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:55,820 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:55,820 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,820 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,820 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:55,877 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/4bf317d10fe2466e9544e172367e9440 2023-03-30 16:54:55,920 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:55,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:55,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:55,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:55,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:55,923 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:55,923 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:55,923 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:55,923 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:55,923 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,923 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,925 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:55,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:55,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:55,925 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,926 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,926 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:55,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:55,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:55,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:55,927 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:55,927 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:55,927 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:55,927 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:55,927 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:55,927 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:55,927 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:55,927 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:55,927 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:55,927 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:55,927 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:55,927 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,927 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:55,927 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:55,929 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,929 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,929 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,929 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:55,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@444e1c7f[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:55,930 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:55,930 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,931 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,931 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:55,931 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:55,931 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:55,931 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,932 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:55,933 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,933 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:55,933 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,933 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:55,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:55,935 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:55,935 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:55,935 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:55,935 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:55,935 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:55,938 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,938 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,938 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:55,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:55,938 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,940 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:55,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,942 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,942 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,942 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:55,942 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:55,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:55,942 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:55,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:55,942 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:55,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:55,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:55,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:55,942 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:55,942 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,942 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:55,942 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:55,942 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:55,942 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:55,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:55,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:56,042 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:56,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:56,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:56,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:56,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:56,046 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:56,046 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:56,046 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:56,046 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:56,046 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,046 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,048 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,048 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:56,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:56,048 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,048 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:56,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:56,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:56,049 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:56,049 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:56,049 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:56,049 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:56,050 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:56,050 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:56,050 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:56,050 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:56,051 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,051 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,052 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:56,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cc0a15b[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:56,053 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:56,053 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,055 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:56,055 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,055 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:56,057 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,057 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:56,057 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,057 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:56,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:56,059 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:56,059 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:56,059 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:56,059 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:56,059 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:56,061 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,061 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,061 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:56,062 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,062 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,064 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,065 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,065 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,065 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:56,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:56,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:56,065 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,065 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:56,065 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,065 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:56,065 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:56,065 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,065 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:56,065 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:56,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:56,165 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:56,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:56,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:56,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:56,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:56,168 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:56,168 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:56,168 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:56,168 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:56,172 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,172 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,181 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,181 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:56,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:56,181 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,181 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:56,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:56,183 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:56,183 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:56,183 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:56,183 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:56,183 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:56,183 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:56,183 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,183 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:56,183 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:56,184 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,185 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,185 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:56,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@408d3e8a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:56,187 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:56,187 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,188 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,188 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:56,188 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:56,188 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:56,188 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,188 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:56,191 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:56,191 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:56,191 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,191 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:56,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:56,193 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:56,193 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:56,193 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:56,193 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:56,193 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:56,195 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,195 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,195 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:56,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:56,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:56,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,196 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,197 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,200 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,200 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,200 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:56,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:56,200 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:56,200 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:56,200 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,200 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,200 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,200 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,200 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:56,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:56,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:56,200 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:56,200 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:56,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:56,286 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/b4786236586b4fd8a55579b045fd4827 2023-03-30 16:54:56,290 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/d7d3b15f87884e8da550a2eac050b8f9 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/d7d3b15f87884e8da550a2eac050b8f9 2023-03-30 16:54:56,294 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/d7d3b15f87884e8da550a2eac050b8f9, entries=200, sequenceid=537, filesize=14.3 K 2023-03-30 16:54:56,295 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/4bf317d10fe2466e9544e172367e9440 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/4bf317d10fe2466e9544e172367e9440 2023-03-30 16:54:56,298 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/4bf317d10fe2466e9544e172367e9440, entries=150, sequenceid=537, filesize=11.9 K 2023-03-30 16:54:56,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:56,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195356294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,299 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/b4786236586b4fd8a55579b045fd4827 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/b4786236586b4fd8a55579b045fd4827 2023-03-30 16:54:56,301 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:56,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:56,302 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/b4786236586b4fd8a55579b045fd4827, entries=150, sequenceid=537, filesize=11.9 K 2023-03-30 16:54:56,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:56,303 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2720c59305e818889603edf8ccad1c2c in 1255ms, sequenceid=537, compaction requested=true 2023-03-30 16:54:56,303 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:56,303 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:56,303 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 16:54:56,304 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 16:54:56,303 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:56,304 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:56,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:56,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:56,304 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:56,304 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:56,304 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:56,304 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:56,304 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,304 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,306 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:56,306 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:56,307 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,307 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:56,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:56,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:56,309 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:56,309 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:56,309 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64120 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 16:54:56,309 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 16:54:56,309 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:56,309 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/B is initiating minor compaction (all files) 2023-03-30 16:54:56,309 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:56,309 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/B in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:56,309 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/A is initiating minor compaction (all files) 2023-03-30 16:54:56,309 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/33fcda4dd5bb4d3b92ebe5f16c3e0db5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/09c3532d8f804d588ad7ad37405aa361, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/0e696543633b4b71bf0a6ae427b35ff0, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/522148798a9a4a6bac587e3cef190350, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/4bf317d10fe2466e9544e172367e9440] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=60.2 K 2023-03-30 16:54:56,309 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:56,309 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/A in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:56,309 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:56,309 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/dbee5f3cdf1e424980b8371a6ca3bfe6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/9db401505d874233995640121d24423e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/ece3c8fed2d54804a4e3a6d545ebfb2a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/c48dc92e18fc4222b9775aebd9415e9b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/d7d3b15f87884e8da550a2eac050b8f9] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=62.6 K 2023-03-30 16:54:56,309 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-30 16:54:56,309 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 33fcda4dd5bb4d3b92ebe5f16c3e0db5, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1680195290907 2023-03-30 16:54:56,309 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:56,309 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:56,310 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:56,310 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:56,310 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting dbee5f3cdf1e424980b8371a6ca3bfe6, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1680195290907 2023-03-30 16:54:56,310 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:56,310 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:56,310 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 09c3532d8f804d588ad7ad37405aa361, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1680195291056 2023-03-30 16:54:56,310 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 9db401505d874233995640121d24423e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1680195291056 2023-03-30 16:54:56,310 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 0e696543633b4b71bf0a6ae427b35ff0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680195292220 2023-03-30 16:54:56,310 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 522148798a9a4a6bac587e3cef190350, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680195292898 2023-03-30 16:54:56,311 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 4bf317d10fe2466e9544e172367e9440, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680195295038 2023-03-30 16:54:56,318 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting ece3c8fed2d54804a4e3a6d545ebfb2a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680195292220 2023-03-30 16:54:56,318 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting c48dc92e18fc4222b9775aebd9415e9b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680195292898 2023-03-30 16:54:56,319 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#B#compaction#256 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:56,319 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting d7d3b15f87884e8da550a2eac050b8f9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680195295038 2023-03-30 16:54:56,326 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/5c1217763f044a1aa00ea0aa9b230181 2023-03-30 16:54:56,333 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#A#compaction#257 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:56,338 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/eae689ac856f4398ac5f25e20f618132 2023-03-30 16:54:56,347 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/ab1f393d420141b3b0885101836d3e23 2023-03-30 16:54:56,352 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/5c1217763f044a1aa00ea0aa9b230181 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/5c1217763f044a1aa00ea0aa9b230181 2023-03-30 16:54:56,357 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/5c1217763f044a1aa00ea0aa9b230181, entries=150, sequenceid=574, filesize=11.9 K 2023-03-30 16:54:56,358 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/eae689ac856f4398ac5f25e20f618132 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/eae689ac856f4398ac5f25e20f618132 2023-03-30 16:54:56,361 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/eae689ac856f4398ac5f25e20f618132, entries=150, sequenceid=574, filesize=11.9 K 2023-03-30 16:54:56,362 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/ab1f393d420141b3b0885101836d3e23 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/ab1f393d420141b3b0885101836d3e23 2023-03-30 16:54:56,366 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/ab1f393d420141b3b0885101836d3e23, entries=150, sequenceid=574, filesize=11.9 K 2023-03-30 16:54:56,367 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 2720c59305e818889603edf8ccad1c2c in 58ms, sequenceid=574, compaction requested=true 2023-03-30 16:54:56,367 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:56,367 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:56,367 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:56,367 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:56,367 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,367 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:56,367 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:56,370 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,370 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,371 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,371 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:56,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@706daa94[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:56,372 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:56,372 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,373 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:56,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:56,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:56,374 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,374 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:56,375 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:56,375 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,375 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,375 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:56,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:56,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:56,378 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:56,378 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:56,378 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:56,378 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:56,378 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:56,379 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,379 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,379 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:56,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:56,380 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,381 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,385 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,385 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,385 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,385 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:56,385 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:56,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:56,385 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,385 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:56,385 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,385 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:56,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:56,385 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:56,385 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:56,385 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:56,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:56,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:56,486 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:56,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:56,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:56,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:56,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:56,489 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:56,489 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:56,489 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:56,489 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:56,489 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,489 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,490 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:56,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:56,490 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,491 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,491 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:56,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:56,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:56,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:56,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:56,492 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:56,492 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:56,492 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:56,492 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:56,492 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:56,493 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,493 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:56,493 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:56,494 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,494 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:56,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44f4ed02[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:56,496 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:56,496 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,497 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,497 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:56,497 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:56,497 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:56,497 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,497 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:56,500 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,500 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:56,500 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:56,500 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:56,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:56,502 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:56,503 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:56,503 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:56,503 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:56,503 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:56,504 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,504 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,504 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,505 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,506 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,508 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:56,508 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:56,508 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:56,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:56,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:56,508 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:56,508 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,509 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:56,509 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:56,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:56,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:56,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,609 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:56,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:56,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:56,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:56,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:56,613 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:56,613 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:56,613 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:56,613 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:56,613 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,613 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,614 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,614 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:56,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:56,615 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,615 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:56,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:56,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:56,616 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:56,617 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:56,617 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:56,617 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:56,617 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:56,617 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:56,617 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,617 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:56,617 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:56,623 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,623 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,623 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,623 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:56,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5da756be[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:56,625 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:56,625 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,626 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,627 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:56,627 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:56,627 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:56,627 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,627 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:56,628 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:56,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:56,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:56,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:56,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:56,631 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:56,632 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:56,632 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:56,632 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:56,632 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:56,633 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,633 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:56,633 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,634 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,635 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,638 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,638 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,638 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,638 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,638 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,638 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:56,638 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:56,638 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:56,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:56,638 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,638 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:56,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:56,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:56,639 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:56,639 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:56,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:56,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:56,732 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/779866a5698d4b93a97a593817a94971 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/779866a5698d4b93a97a593817a94971 2023-03-30 16:54:56,736 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/B of 2720c59305e818889603edf8ccad1c2c into 779866a5698d4b93a97a593817a94971(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:56,736 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:56,736 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/B, priority=11, startTime=1680195296303; duration=0sec 2023-03-30 16:54:56,737 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:56,737 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-30 16:54:56,738 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73866 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-30 16:54:56,738 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/C is initiating minor compaction (all files) 2023-03-30 16:54:56,738 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/C in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:56,738 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/d4a88e39c4ce46f996af350011242962, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/8ebfd50998d1486d851ddad2458334c0, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/b0b6eceff6414c4a858c8b191f1f000c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/9b679f2a311b432ba1243bda12e1a03f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/b4786236586b4fd8a55579b045fd4827, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/ab1f393d420141b3b0885101836d3e23] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=72.1 K 2023-03-30 16:54:56,739 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:56,739 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting d4a88e39c4ce46f996af350011242962, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1680195290907 2023-03-30 16:54:56,739 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 8ebfd50998d1486d851ddad2458334c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1680195291056 2023-03-30 16:54:56,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:56,740 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting b0b6eceff6414c4a858c8b191f1f000c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680195292220 2023-03-30 16:54:56,740 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 9b679f2a311b432ba1243bda12e1a03f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680195292898 2023-03-30 16:54:56,741 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting b4786236586b4fd8a55579b045fd4827, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680195295038 2023-03-30 16:54:56,741 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting ab1f393d420141b3b0885101836d3e23, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680195295167 2023-03-30 16:54:56,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:56,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:56,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:56,743 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:56,743 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:56,743 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:56,743 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:56,743 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,743 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,744 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/0be19e1c87c8499db4883a37fc975399 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/0be19e1c87c8499db4883a37fc975399 2023-03-30 16:54:56,744 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:56,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:56,744 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,745 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,745 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:56,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:56,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:56,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:56,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:56,746 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:56,746 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:56,746 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:56,746 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:56,747 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:56,747 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:56,747 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:56,747 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:56,747 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:56,747 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:56,747 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:56,747 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,747 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:56,747 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:56,750 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/A of 2720c59305e818889603edf8ccad1c2c into 0be19e1c87c8499db4883a37fc975399(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:56,750 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:56,750 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/A, priority=11, startTime=1680195296303; duration=0sec 2023-03-30 16:54:56,750 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:56,752 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,752 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,752 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,752 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:56,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:56,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74775aee[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:56,753 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:56,753 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,754 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,755 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:56,755 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:56,755 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,755 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:56,755 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:56,756 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#C#compaction#260 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:56,757 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,757 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:56,757 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:56,757 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:56,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:56,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:56,761 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:56,761 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:56,761 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:56,761 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:56,761 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:56,762 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,763 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,763 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:56,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:56,763 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,765 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,767 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,768 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:56,768 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,768 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:56,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:56,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:56,768 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:56,768 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,768 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,768 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,768 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,768 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:56,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:56,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:56,768 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:56,768 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:56,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:56,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:56,868 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:56,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:56,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:56,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:56,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:56,876 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:56,876 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:56,876 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:56,876 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:56,876 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,876 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,877 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:56,878 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:56,878 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,878 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:56,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:56,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:56,879 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:56,879 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:56,879 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:56,880 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:56,880 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:56,880 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:56,880 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:56,880 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:56,880 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:56,880 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:56,880 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:56,880 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,880 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:56,880 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:56,883 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,883 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,883 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,883 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:56,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:56,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@695367fe[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:56,885 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:56,885 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,886 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,887 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:56,887 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:56,887 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:56,887 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,887 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:56,888 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,888 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:56,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,888 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:56,888 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:56,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:56,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:56,890 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:56,890 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:56,890 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:56,890 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:56,890 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:56,892 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,892 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:56,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:56,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,892 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:56,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:56,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:56,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:56,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:56,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,893 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:56,894 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,898 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:56,898 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:56,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:56,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:56,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:56,898 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:56,898 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:56,898 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:56,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:56,898 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:56,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:56,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:56,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:56,998 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:56,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:57,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:57,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:57,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:57,002 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:57,002 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:57,002 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:57,002 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:57,002 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,002 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,003 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:57,003 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:57,004 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,004 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:57,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:57,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:57,005 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:57,005 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:57,005 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:57,005 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:57,005 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:57,005 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:57,005 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:57,006 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:57,006 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:57,006 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:57,006 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:57,006 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,006 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:57,006 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:57,007 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:57,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:57,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:57,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a2c0b2b[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:57,009 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:57,009 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,010 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,010 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:57,010 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:57,010 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:57,010 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,010 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:57,015 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,015 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:57,015 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:57,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:57,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:57,015 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:57,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:57,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:57,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:57,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:57,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:57,018 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:57,018 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:57,018 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:57,018 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:57,018 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:57,019 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,019 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:57,019 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:57,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:57,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,020 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:57,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:57,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:57,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,021 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:57,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:57,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:57,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:57,024 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:57,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,024 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:57,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:57,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:57,024 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:57,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:57,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:57,124 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:57,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:57,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:57,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:57,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:57,130 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:57,130 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:57,130 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:57,130 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:57,131 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,131 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,132 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:57,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:57,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:57,133 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,133 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,133 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:57,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:57,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:57,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:57,134 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:57,134 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:57,134 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:57,136 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:57,136 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:57,136 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:57,136 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:57,136 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:57,136 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:57,136 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:57,136 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:57,136 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,136 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:57,136 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:57,139 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,139 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:57,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:57,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:57,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:57,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:57,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:57,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dd9ae76[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:57,141 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:57,142 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,143 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,143 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:57,143 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:57,143 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:57,143 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,143 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:57,144 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,144 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:57,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:57,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:57,144 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:57,144 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:57,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:57,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:57,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:57,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:57,148 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:57,148 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:57,148 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:57,148 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:57,148 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:57,150 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,150 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,150 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:57,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,151 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,153 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,158 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:57,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:57,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:57,158 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,159 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,159 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,159 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:57,159 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:57,159 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,159 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:57,159 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,159 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:57,159 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,159 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:57,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:57,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:57,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:57,170 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/85b5311d5ec545aca4eebb4e5e1bc404 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/85b5311d5ec545aca4eebb4e5e1bc404 2023-03-30 16:54:57,174 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/C of 2720c59305e818889603edf8ccad1c2c into 85b5311d5ec545aca4eebb4e5e1bc404(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:57,174 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:57,174 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/C, priority=10, startTime=1680195296304; duration=0sec 2023-03-30 16:54:57,174 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:57,259 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:57,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:57,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:57,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:57,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:57,262 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:57,262 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:57,262 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:57,262 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:57,263 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,263 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,264 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:57,264 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:57,264 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,264 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:57,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:57,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:57,266 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:57,266 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:57,266 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:57,266 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:57,266 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:57,266 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:57,266 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:57,266 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:57,266 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:57,266 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:57,266 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:57,266 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,266 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:57,266 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:57,268 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:57,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:57,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:57,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:57,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23b72909[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:57,270 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:57,270 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,271 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,272 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:57,272 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:57,272 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:57,272 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,272 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:57,274 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:57,274 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:57,274 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,274 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:57,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:57,276 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:57,277 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:57,277 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:57,277 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:57,277 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:57,278 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,278 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,278 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:57,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:57,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:57,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:57,279 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:57,280 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,283 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,283 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,283 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,283 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,283 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:57,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:57,283 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,283 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:57,283 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:57,283 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:57,283 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:57,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:57,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:57,283 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:57,283 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:57,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:57,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:57,312 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:54:57,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:57,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:57,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:57,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:57,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:57,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:54:57,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195297349, max timestamp=1680195297357Num uniques 859; 2023-03-30 16:54:57,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:57,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195297349, max timestamp=1680195297357Num uniques 859; 2023-03-30 16:54:57,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195297349, max timestamp=1680195297357Num uniques 859; 2023-03-30 16:54:57,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:57,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:54:57,383 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:57,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:57,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:57,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:57,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:57,386 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:57,386 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:57,386 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:57,386 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:57,387 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,387 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,388 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:57,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:57,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:57,388 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,388 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,388 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:57,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:57,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,389 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:57,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:57,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:57,390 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:57,390 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:57,390 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:57,390 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:57,390 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:57,390 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:57,390 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:57,390 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:57,390 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:57,390 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:57,390 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:57,390 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,391 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:57,391 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:57,393 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,393 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:57,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:57,393 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,393 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:57,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:57,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:57,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:57,394 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:57,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5129c616[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:57,394 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,395 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,396 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:57,396 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:57,396 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:57,396 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,396 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:57,397 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,397 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:57,397 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:57,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:57,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:57,397 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:57,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:57,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:57,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:57,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:57,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:57,399 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:57,399 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:57,400 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:57,400 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:57,400 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:57,405 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,405 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,405 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:57,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:57,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:57,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:57,406 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:57,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,407 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,409 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,409 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,409 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,409 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:57,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:57,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:57,409 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,409 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:57,409 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:57,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:57,409 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:57,409 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:57,409 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:57,409 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:57,409 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:57,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:57,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:57,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 434 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195357410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,510 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:57,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:57,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:54:57,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:54:57,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:54:57,513 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:54:57,513 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:57,513 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:54:57,513 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:54:57,513 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,513 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:57,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 436 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195357513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,515 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,515 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:57,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:57,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:57,515 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,515 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:54:57,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:54:57,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:54:57,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:54:57,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:54:57,516 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:54:57,516 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:54:57,516 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:54:57,518 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:54:57,518 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:54:57,518 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:57,518 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. started... 2023-03-30 16:54:57,518 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. as already flushing 2023-03-30 16:54:57,518 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:57,518 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:54:57,518 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:54:57,518 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,518 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:54:57,518 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:54:57,520 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,520 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:57,520 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,520 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:54:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:57,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:54:57,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bb6ea4b[Count = 0] remaining members to acquire global barrier 2023-03-30 16:54:57,521 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:54:57,522 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,524 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,524 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:54:57,524 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:54:57,524 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:54:57,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:54:57,537 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,537 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:54:57,537 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:57,537 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:54:57,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:57,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:57,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:54:57,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:54:57,542 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:54:57,542 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:54:57,542 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:54:57,542 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:54:57,542 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:54:57,544 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,544 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:54:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:54:57,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,544 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:54:57,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:57,545 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:54:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,546 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:54:57,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:54:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,548 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:54:57,548 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:54:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:54:57,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:54:57,548 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:54:57,548 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,548 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:54:57,549 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,549 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,549 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:54:57,549 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:54:57,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:54:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:54:57,552 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:54:57,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:54:57,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:54:57,552 DEBUG [Thread-1453] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:54:57,552 DEBUG [Thread-1453] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:54:57,636 DEBUG [Thread-1454] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b39d5b8 to 127.0.0.1:58435 2023-03-30 16:54:57,636 DEBUG [Thread-1454] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:57,637 DEBUG [Thread-1456] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x44f9bd75 to 127.0.0.1:58435 2023-03-30 16:54:57,637 DEBUG [Thread-1456] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:57,638 DEBUG [Thread-1458] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e5b1c14 to 127.0.0.1:58435 2023-03-30 16:54:57,638 DEBUG [Thread-1458] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:57,640 DEBUG [Thread-1462] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x681ace6c to 127.0.0.1:58435 2023-03-30 16:54:57,640 DEBUG [Thread-1462] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:57,641 DEBUG [Thread-1460] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f29c1c4 to 127.0.0.1:58435 2023-03-30 16:54:57,641 DEBUG [Thread-1460] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:57,652 DEBUG [Thread-1453] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:54:57,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:54:57,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:57,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 438 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195357715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:57,719 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/2720edfd5a44409c8c746a9d4dd974c1 2023-03-30 16:54:58,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:58,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 440 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195358016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:58,127 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/38be34fa8cfc4fc3898febe36508c45b 2023-03-30 16:54:58,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:54:58,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 442 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34430 deadline: 1680195358519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:58,536 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/dc2f12d62fc040638fb0733de31f6749 2023-03-30 16:54:58,540 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/2720edfd5a44409c8c746a9d4dd974c1 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/2720edfd5a44409c8c746a9d4dd974c1 2023-03-30 16:54:58,543 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/2720edfd5a44409c8c746a9d4dd974c1, entries=200, sequenceid=594, filesize=14.3 K 2023-03-30 16:54:58,543 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/38be34fa8cfc4fc3898febe36508c45b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/38be34fa8cfc4fc3898febe36508c45b 2023-03-30 16:54:58,546 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/38be34fa8cfc4fc3898febe36508c45b, entries=150, sequenceid=594, filesize=11.9 K 2023-03-30 16:54:58,547 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/dc2f12d62fc040638fb0733de31f6749 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/dc2f12d62fc040638fb0733de31f6749 2023-03-30 16:54:58,550 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/dc2f12d62fc040638fb0733de31f6749, entries=150, sequenceid=594, filesize=11.9 K 2023-03-30 16:54:58,551 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2720c59305e818889603edf8ccad1c2c in 1239ms, sequenceid=594, compaction requested=true 2023-03-30 16:54:58,551 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:58,551 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:58,551 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:54:58,551 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:58,551 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:54:58,551 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:58,552 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39918 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:54:58,552 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:54:58,552 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/A is initiating minor compaction (all files) 2023-03-30 16:54:58,552 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 2720c59305e818889603edf8ccad1c2c/B is initiating minor compaction (all files) 2023-03-30 16:54:58,552 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/A in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:58,552 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2720c59305e818889603edf8ccad1c2c/B in TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:54:58,552 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/0be19e1c87c8499db4883a37fc975399, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/5c1217763f044a1aa00ea0aa9b230181, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/2720edfd5a44409c8c746a9d4dd974c1] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=39.0 K 2023-03-30 16:54:58,552 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/779866a5698d4b93a97a593817a94971, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/eae689ac856f4398ac5f25e20f618132, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/38be34fa8cfc4fc3898febe36508c45b] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp, totalSize=36.6 K 2023-03-30 16:54:58,552 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 0be19e1c87c8499db4883a37fc975399, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680195295038 2023-03-30 16:54:58,552 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 779866a5698d4b93a97a593817a94971, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680195295038 2023-03-30 16:54:58,553 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 5c1217763f044a1aa00ea0aa9b230181, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680195295167 2023-03-30 16:54:58,553 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting eae689ac856f4398ac5f25e20f618132, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1680195295167 2023-03-30 16:54:58,553 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 2720edfd5a44409c8c746a9d4dd974c1, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1680195297306 2023-03-30 16:54:58,553 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 38be34fa8cfc4fc3898febe36508c45b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1680195297306 2023-03-30 16:54:58,558 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#A#compaction#264 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:58,558 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 2720c59305e818889603edf8ccad1c2c#B#compaction#265 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:54:58,966 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/7310488436d544cab20c92fcbfb05a92 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/7310488436d544cab20c92fcbfb05a92 2023-03-30 16:54:58,966 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/b5fca82aca314125af27344d7277bf51 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/b5fca82aca314125af27344d7277bf51 2023-03-30 16:54:58,969 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/A of 2720c59305e818889603edf8ccad1c2c into 7310488436d544cab20c92fcbfb05a92(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:58,969 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2720c59305e818889603edf8ccad1c2c/B of 2720c59305e818889603edf8ccad1c2c into b5fca82aca314125af27344d7277bf51(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:54:58,969 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:58,969 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:54:58,969 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/A, priority=13, startTime=1680195298551; duration=0sec 2023-03-30 16:54:58,969 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c., storeName=2720c59305e818889603edf8ccad1c2c/B, priority=13, startTime=1680195298551; duration=0sec 2023-03-30 16:54:58,970 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:54:58,970 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:54:58,970 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-30 16:54:58,970 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-30 16:54:58,970 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-30 16:54:58,970 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. because compaction request was cancelled 2023-03-30 16:54:59,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:54:59,527 DEBUG [Thread-1449] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10b79c2c to 127.0.0.1:58435 2023-03-30 16:54:59,527 DEBUG [Thread-1449] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:59,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680195297388, max timestamp=1680195299525Num uniques 1018; 2023-03-30 16:54:59,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-30 16:54:59,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680195297388, max timestamp=1680195299525Num uniques 1018; 2023-03-30 16:54:59,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-30 16:54:59,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680195297388, max timestamp=1680195299525Num uniques 1018; 2023-03-30 16:54:59,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-30 16:54:59,529 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=26.84 KB heapSize=65.39 KB 2023-03-30 16:54:59,529 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:54:59,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:59,529 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:54:59,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:59,529 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:54:59,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:54:59,681 DEBUG [Thread-1451] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a5a81fa to 127.0.0.1:58435 2023-03-30 16:54:59,681 DEBUG [Thread-1451] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:59,869 DEBUG [Thread-1445] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x53921272 to 127.0.0.1:58435 2023-03-30 16:54:59,869 DEBUG [Thread-1445] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:59,885 DEBUG [Thread-1447] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13544701 to 127.0.0.1:58435 2023-03-30 16:54:59,885 DEBUG [Thread-1447] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:59,916 DEBUG [Thread-1443] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x47772cff to 127.0.0.1:58435 2023-03-30 16:54:59,916 DEBUG [Thread-1443] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:59,916 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-30 16:54:59,916 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 30 2023-03-30 16:54:59,916 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 25 2023-03-30 16:54:59,916 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 26 2023-03-30 16:54:59,916 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 323 2023-03-30 16:54:59,916 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 119 2023-03-30 16:54:59,916 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-30 16:54:59,916 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(398): read 5580 2023-03-30 16:54:59,916 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(398): read 5494 2023-03-30 16:54:59,917 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(398): read 5321 2023-03-30 16:54:59,917 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(398): read 5623 2023-03-30 16:54:59,917 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(398): read 5497 2023-03-30 16:54:59,917 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-30 16:54:59,917 INFO [Listener at localhost/38075] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-30 16:54:59,917 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23083cf2 to 127.0.0.1:58435 2023-03-30 16:54:59,917 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:54:59,917 INFO [Listener at localhost/38075] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-30 16:54:59,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-30 16:54:59,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-30 16:54:59,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-30 16:54:59,921 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195299921"}]},"ts":"1680195299921"} 2023-03-30 16:54:59,922 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-30 16:54:59,924 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-30 16:54:59,925 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2720c59305e818889603edf8ccad1c2c, UNASSIGN}] 2023-03-30 16:54:59,926 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2720c59305e818889603edf8ccad1c2c, UNASSIGN 2023-03-30 16:54:59,927 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=2720c59305e818889603edf8ccad1c2c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:54:59,927 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195299927"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680195299927"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680195299927"}]},"ts":"1680195299927"} 2023-03-30 16:54:59,927 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285}] 2023-03-30 16:54:59,936 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/914d819760914bf78cf57f7c0c2b5009 2023-03-30 16:55:00,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-30 16:55:00,079 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:55:00,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2720c59305e818889603edf8ccad1c2c, disabling compactions & flushes 2023-03-30 16:55:00,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:55:00,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-30 16:55:00,345 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/acd46be900bc47ed87649142265ebd8c 2023-03-30 16:55:00,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-30 16:55:00,753 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/966d7e4099184ada91f9feaefa99a6a1 2023-03-30 16:55:00,757 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/914d819760914bf78cf57f7c0c2b5009 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/914d819760914bf78cf57f7c0c2b5009 2023-03-30 16:55:00,760 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/914d819760914bf78cf57f7c0c2b5009, entries=150, sequenceid=634, filesize=11.9 K 2023-03-30 16:55:00,760 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/acd46be900bc47ed87649142265ebd8c as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/acd46be900bc47ed87649142265ebd8c 2023-03-30 16:55:00,764 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/acd46be900bc47ed87649142265ebd8c, entries=150, sequenceid=634, filesize=11.9 K 2023-03-30 16:55:00,765 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/966d7e4099184ada91f9feaefa99a6a1 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/966d7e4099184ada91f9feaefa99a6a1 2023-03-30 16:55:00,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/966d7e4099184ada91f9feaefa99a6a1, entries=150, sequenceid=634, filesize=11.9 K 2023-03-30 16:55:00,768 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~65.34 KB/66912, currentSize=26.84 KB/27480 for 2720c59305e818889603edf8ccad1c2c in 1239ms, sequenceid=634, compaction requested=true 2023-03-30 16:55:00,769 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:55:00,769 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:55:00,769 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:00,769 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. because compaction request was cancelled 2023-03-30 16:55:00,769 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. because compaction request was cancelled 2023-03-30 16:55:00,769 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:00,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:55:00,769 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:00,769 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. because compaction request was cancelled 2023-03-30 16:55:00,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. after waiting 0 ms 2023-03-30 16:55:00,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:55:00,769 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 2720c59305e818889603edf8ccad1c2c 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-30 16:55:00,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=A 2023-03-30 16:55:00,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:00,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=B 2023-03-30 16:55:00,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:00,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2720c59305e818889603edf8ccad1c2c, store=C 2023-03-30 16:55:00,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:01,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-30 16:55:01,175 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/af2a32a2e3e74971a37c0fedb7ce4566 2023-03-30 16:55:01,185 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/d27bcf6a5e33494ea9f1c76ea2c0933b 2023-03-30 16:55:01,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/3994808d145e4caeb4d3693a8acffba2 2023-03-30 16:55:01,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/A/af2a32a2e3e74971a37c0fedb7ce4566 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/af2a32a2e3e74971a37c0fedb7ce4566 2023-03-30 16:55:01,199 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/af2a32a2e3e74971a37c0fedb7ce4566, entries=100, sequenceid=641, filesize=9.5 K 2023-03-30 16:55:01,200 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/B/d27bcf6a5e33494ea9f1c76ea2c0933b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/d27bcf6a5e33494ea9f1c76ea2c0933b 2023-03-30 16:55:01,203 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/d27bcf6a5e33494ea9f1c76ea2c0933b, entries=100, sequenceid=641, filesize=9.5 K 2023-03-30 16:55:01,204 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/.tmp/C/3994808d145e4caeb4d3693a8acffba2 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/3994808d145e4caeb4d3693a8acffba2 2023-03-30 16:55:01,206 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/3994808d145e4caeb4d3693a8acffba2, entries=100, sequenceid=641, filesize=9.5 K 2023-03-30 16:55:01,207 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 2720c59305e818889603edf8ccad1c2c in 438ms, sequenceid=641, compaction requested=true 2023-03-30 16:55:01,227 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/218eaa2da37e4e22a85fc59f18e2b169, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/46a5965e09a8423583db24b7d60b13f3, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/8476ffb1dffa44cc8f94ba9a50967f09, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/76cd012a853c44118133f59a6fa97f1a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/327b00dca9724ea9adc176ea2506243d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/5d53e13bf81a48e38d96b0363ed68aff, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/1c29b57a95d945f9a7e3a25f2be4cb49, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/90c34b1eff734b68a287c4fa65afb21b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/b8f1d38a233646e98a4e50d3569b9194, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/d9165028db664ef1a695a23369089a2b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/da89c421a2ba41ba9a1b856f1d904f91, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/bc80ad3a029748128b606cb83f37eca4, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/4751def338304e618029d8131df531fb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/24ba73a3dddf42dd89a691db0bedcfc0, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/fc53863e0b6a467d9ba1ed3c8fa591d8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/0757e7cf841e4249ab09d837230af77d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/2238984969604415bfeea5193e12cc2c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/85946b340f584ac28cbe3e23d06d0bde, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/80ada46f04f84f739f50f50ffdc822ef, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/d4ba6ca8b0db4a83938a81538f87d19d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/58b269f1a7c9442880799a55a7ab4d03, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/3651f93c436f44088d019d31918ba9a4, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/dbee5f3cdf1e424980b8371a6ca3bfe6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/9db401505d874233995640121d24423e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/ece3c8fed2d54804a4e3a6d545ebfb2a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/c48dc92e18fc4222b9775aebd9415e9b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/d7d3b15f87884e8da550a2eac050b8f9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/0be19e1c87c8499db4883a37fc975399, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/5c1217763f044a1aa00ea0aa9b230181, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/2720edfd5a44409c8c746a9d4dd974c1] to archive 2023-03-30 16:55:01,228 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 16:55:01,229 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/218eaa2da37e4e22a85fc59f18e2b169 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/218eaa2da37e4e22a85fc59f18e2b169 2023-03-30 16:55:01,230 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/46a5965e09a8423583db24b7d60b13f3 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/46a5965e09a8423583db24b7d60b13f3 2023-03-30 16:55:01,231 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/8476ffb1dffa44cc8f94ba9a50967f09 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/8476ffb1dffa44cc8f94ba9a50967f09 2023-03-30 16:55:01,232 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/76cd012a853c44118133f59a6fa97f1a to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/76cd012a853c44118133f59a6fa97f1a 2023-03-30 16:55:01,233 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/327b00dca9724ea9adc176ea2506243d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/327b00dca9724ea9adc176ea2506243d 2023-03-30 16:55:01,234 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/5d53e13bf81a48e38d96b0363ed68aff to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/5d53e13bf81a48e38d96b0363ed68aff 2023-03-30 16:55:01,235 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/1c29b57a95d945f9a7e3a25f2be4cb49 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/1c29b57a95d945f9a7e3a25f2be4cb49 2023-03-30 16:55:01,236 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/90c34b1eff734b68a287c4fa65afb21b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/90c34b1eff734b68a287c4fa65afb21b 2023-03-30 16:55:01,237 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/b8f1d38a233646e98a4e50d3569b9194 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/b8f1d38a233646e98a4e50d3569b9194 2023-03-30 16:55:01,237 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/d9165028db664ef1a695a23369089a2b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/d9165028db664ef1a695a23369089a2b 2023-03-30 16:55:01,238 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/da89c421a2ba41ba9a1b856f1d904f91 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/da89c421a2ba41ba9a1b856f1d904f91 2023-03-30 16:55:01,239 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/bc80ad3a029748128b606cb83f37eca4 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/bc80ad3a029748128b606cb83f37eca4 2023-03-30 16:55:01,240 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/4751def338304e618029d8131df531fb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/4751def338304e618029d8131df531fb 2023-03-30 16:55:01,241 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/24ba73a3dddf42dd89a691db0bedcfc0 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/24ba73a3dddf42dd89a691db0bedcfc0 2023-03-30 16:55:01,242 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/fc53863e0b6a467d9ba1ed3c8fa591d8 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/fc53863e0b6a467d9ba1ed3c8fa591d8 2023-03-30 16:55:01,242 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/0757e7cf841e4249ab09d837230af77d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/0757e7cf841e4249ab09d837230af77d 2023-03-30 16:55:01,243 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/2238984969604415bfeea5193e12cc2c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/2238984969604415bfeea5193e12cc2c 2023-03-30 16:55:01,244 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/85946b340f584ac28cbe3e23d06d0bde to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/85946b340f584ac28cbe3e23d06d0bde 2023-03-30 16:55:01,245 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/80ada46f04f84f739f50f50ffdc822ef to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/80ada46f04f84f739f50f50ffdc822ef 2023-03-30 16:55:01,246 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/d4ba6ca8b0db4a83938a81538f87d19d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/d4ba6ca8b0db4a83938a81538f87d19d 2023-03-30 16:55:01,247 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/58b269f1a7c9442880799a55a7ab4d03 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/58b269f1a7c9442880799a55a7ab4d03 2023-03-30 16:55:01,248 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/3651f93c436f44088d019d31918ba9a4 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/3651f93c436f44088d019d31918ba9a4 2023-03-30 16:55:01,249 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/dbee5f3cdf1e424980b8371a6ca3bfe6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/dbee5f3cdf1e424980b8371a6ca3bfe6 2023-03-30 16:55:01,250 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/9db401505d874233995640121d24423e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/9db401505d874233995640121d24423e 2023-03-30 16:55:01,251 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/ece3c8fed2d54804a4e3a6d545ebfb2a to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/ece3c8fed2d54804a4e3a6d545ebfb2a 2023-03-30 16:55:01,252 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/c48dc92e18fc4222b9775aebd9415e9b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/c48dc92e18fc4222b9775aebd9415e9b 2023-03-30 16:55:01,253 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/d7d3b15f87884e8da550a2eac050b8f9 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/d7d3b15f87884e8da550a2eac050b8f9 2023-03-30 16:55:01,253 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/0be19e1c87c8499db4883a37fc975399 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/0be19e1c87c8499db4883a37fc975399 2023-03-30 16:55:01,254 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/5c1217763f044a1aa00ea0aa9b230181 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/5c1217763f044a1aa00ea0aa9b230181 2023-03-30 16:55:01,255 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/2720edfd5a44409c8c746a9d4dd974c1 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/2720edfd5a44409c8c746a9d4dd974c1 2023-03-30 16:55:01,258 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/7d907bd4b7814d2c97c218321e9ec277, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/5e2289818caf445a9b1e534b587d4b9b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/ffa92e7f50d24f9e891eb9648ac10009, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/642c29a0c2da44d8b977b0cfba41268f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/8eaefb7daf8b447783109ff29e603c3c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/fe96aec2f5e64fab939bce6f31023ca1, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/62c78c882a27414f9e000303debb4a3e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/36f56cd004a84956bba20be219503174, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/95c7666bbdaa4061a26b21a0bc55ccd6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/23c8e1b6f29141df9949ff18222b7ccb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/bb65618f8ff5427f9ca9afe61087cb79, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/4b1345e537df442faa248733acc473ed, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/ed6b59760cf64407b6e48f96a19af830, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/ee571cc789d643e5b2b08d6198626f99, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/6769e481c46f4ec18ec2e84dc17beb86, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/4f8a8ac4a5c849f592198486a398e066, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/26122dd9c9e44f3bbe5da85a2b6bf639, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/513d4cedfc074435b31740a80df5866e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/87cb4ea383b04886a89fb54142018b5b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/9fed90faffb641518a0cfec9f97f3b42, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/946467c91efc4940b73d2ac3ae49eeed, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/33fcda4dd5bb4d3b92ebe5f16c3e0db5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/e7368180d4a74b6bbe9ee78b29f1c693, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/09c3532d8f804d588ad7ad37405aa361, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/0e696543633b4b71bf0a6ae427b35ff0, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/522148798a9a4a6bac587e3cef190350, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/779866a5698d4b93a97a593817a94971, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/4bf317d10fe2466e9544e172367e9440, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/eae689ac856f4398ac5f25e20f618132, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/38be34fa8cfc4fc3898febe36508c45b] to archive 2023-03-30 16:55:01,259 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 16:55:01,260 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/7d907bd4b7814d2c97c218321e9ec277 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/7d907bd4b7814d2c97c218321e9ec277 2023-03-30 16:55:01,261 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/5e2289818caf445a9b1e534b587d4b9b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/5e2289818caf445a9b1e534b587d4b9b 2023-03-30 16:55:01,261 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/ffa92e7f50d24f9e891eb9648ac10009 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/ffa92e7f50d24f9e891eb9648ac10009 2023-03-30 16:55:01,262 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/642c29a0c2da44d8b977b0cfba41268f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/642c29a0c2da44d8b977b0cfba41268f 2023-03-30 16:55:01,263 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/8eaefb7daf8b447783109ff29e603c3c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/8eaefb7daf8b447783109ff29e603c3c 2023-03-30 16:55:01,264 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/fe96aec2f5e64fab939bce6f31023ca1 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/fe96aec2f5e64fab939bce6f31023ca1 2023-03-30 16:55:01,265 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/62c78c882a27414f9e000303debb4a3e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/62c78c882a27414f9e000303debb4a3e 2023-03-30 16:55:01,266 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/36f56cd004a84956bba20be219503174 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/36f56cd004a84956bba20be219503174 2023-03-30 16:55:01,266 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/95c7666bbdaa4061a26b21a0bc55ccd6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/95c7666bbdaa4061a26b21a0bc55ccd6 2023-03-30 16:55:01,267 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/23c8e1b6f29141df9949ff18222b7ccb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/23c8e1b6f29141df9949ff18222b7ccb 2023-03-30 16:55:01,268 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/bb65618f8ff5427f9ca9afe61087cb79 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/bb65618f8ff5427f9ca9afe61087cb79 2023-03-30 16:55:01,269 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/4b1345e537df442faa248733acc473ed to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/4b1345e537df442faa248733acc473ed 2023-03-30 16:55:01,270 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/ed6b59760cf64407b6e48f96a19af830 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/ed6b59760cf64407b6e48f96a19af830 2023-03-30 16:55:01,270 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/ee571cc789d643e5b2b08d6198626f99 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/ee571cc789d643e5b2b08d6198626f99 2023-03-30 16:55:01,271 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/6769e481c46f4ec18ec2e84dc17beb86 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/6769e481c46f4ec18ec2e84dc17beb86 2023-03-30 16:55:01,272 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/4f8a8ac4a5c849f592198486a398e066 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/4f8a8ac4a5c849f592198486a398e066 2023-03-30 16:55:01,273 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/26122dd9c9e44f3bbe5da85a2b6bf639 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/26122dd9c9e44f3bbe5da85a2b6bf639 2023-03-30 16:55:01,273 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/513d4cedfc074435b31740a80df5866e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/513d4cedfc074435b31740a80df5866e 2023-03-30 16:55:01,274 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/87cb4ea383b04886a89fb54142018b5b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/87cb4ea383b04886a89fb54142018b5b 2023-03-30 16:55:01,275 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/9fed90faffb641518a0cfec9f97f3b42 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/9fed90faffb641518a0cfec9f97f3b42 2023-03-30 16:55:01,276 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/946467c91efc4940b73d2ac3ae49eeed to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/946467c91efc4940b73d2ac3ae49eeed 2023-03-30 16:55:01,277 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/33fcda4dd5bb4d3b92ebe5f16c3e0db5 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/33fcda4dd5bb4d3b92ebe5f16c3e0db5 2023-03-30 16:55:01,278 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/e7368180d4a74b6bbe9ee78b29f1c693 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/e7368180d4a74b6bbe9ee78b29f1c693 2023-03-30 16:55:01,279 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/09c3532d8f804d588ad7ad37405aa361 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/09c3532d8f804d588ad7ad37405aa361 2023-03-30 16:55:01,279 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/0e696543633b4b71bf0a6ae427b35ff0 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/0e696543633b4b71bf0a6ae427b35ff0 2023-03-30 16:55:01,280 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/522148798a9a4a6bac587e3cef190350 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/522148798a9a4a6bac587e3cef190350 2023-03-30 16:55:01,281 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/779866a5698d4b93a97a593817a94971 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/779866a5698d4b93a97a593817a94971 2023-03-30 16:55:01,282 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/4bf317d10fe2466e9544e172367e9440 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/4bf317d10fe2466e9544e172367e9440 2023-03-30 16:55:01,283 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/eae689ac856f4398ac5f25e20f618132 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/eae689ac856f4398ac5f25e20f618132 2023-03-30 16:55:01,284 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/38be34fa8cfc4fc3898febe36508c45b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/38be34fa8cfc4fc3898febe36508c45b 2023-03-30 16:55:01,286 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/9862a76ae2074d8b95a2d80e09e8cbf2, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/e685969677f64ba5b9ab3171f3b045fc, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/77f9182a60df4ebf822dc904985d8f54, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/529c276af5b74d6bb6915deb5259aab8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/ad841d7afa9644368cec92b6a941cd93, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/a0ad5b4ae778489aa7de2932b2c92c16, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/3a85f5f248e54cf08b709d9b1647f542, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/37bc30b9f5d544a9b76ffdbc6404bbae, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/59206a349e2b4fedb10edb34456e3493, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/1babef462c554755a8b62fb79c59ebe5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/87a67fb20c2245f1ad933ec7b8bdbb47, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/cce8467d74d242a0863dd45913186f5a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/7457f10778524d9fb0f88e4fe996d4c7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/7777c4885d3c473887073a86e17a4388, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/3be67ce6778d4a37934e3a5d195361bf, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/d912324bc74b479ba8fa9d6bd2e5eb80, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/b0cf6884064e4d24a71e7938fc984a67, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/a9205a1bb7fc4c8685bc0a73e3337cc7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/e79458c7b7254fa79f9f0e996bd3bbf6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/83ae7aa098ad4ac480a69dfaf732786f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/0746da17b0844507a331b3ff81e2ba2a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/d4a88e39c4ce46f996af350011242962, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/149f2accf2a1440d93186e1d0ee7dfe3, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/8ebfd50998d1486d851ddad2458334c0, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/b0b6eceff6414c4a858c8b191f1f000c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/9b679f2a311b432ba1243bda12e1a03f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/b4786236586b4fd8a55579b045fd4827, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/ab1f393d420141b3b0885101836d3e23] to archive 2023-03-30 16:55:01,287 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 16:55:01,288 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/9862a76ae2074d8b95a2d80e09e8cbf2 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/9862a76ae2074d8b95a2d80e09e8cbf2 2023-03-30 16:55:01,289 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/e685969677f64ba5b9ab3171f3b045fc to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/e685969677f64ba5b9ab3171f3b045fc 2023-03-30 16:55:01,290 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/77f9182a60df4ebf822dc904985d8f54 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/77f9182a60df4ebf822dc904985d8f54 2023-03-30 16:55:01,291 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/529c276af5b74d6bb6915deb5259aab8 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/529c276af5b74d6bb6915deb5259aab8 2023-03-30 16:55:01,292 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/ad841d7afa9644368cec92b6a941cd93 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/ad841d7afa9644368cec92b6a941cd93 2023-03-30 16:55:01,293 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/a0ad5b4ae778489aa7de2932b2c92c16 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/a0ad5b4ae778489aa7de2932b2c92c16 2023-03-30 16:55:01,293 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/3a85f5f248e54cf08b709d9b1647f542 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/3a85f5f248e54cf08b709d9b1647f542 2023-03-30 16:55:01,294 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/37bc30b9f5d544a9b76ffdbc6404bbae to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/37bc30b9f5d544a9b76ffdbc6404bbae 2023-03-30 16:55:01,295 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/59206a349e2b4fedb10edb34456e3493 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/59206a349e2b4fedb10edb34456e3493 2023-03-30 16:55:01,296 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/1babef462c554755a8b62fb79c59ebe5 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/1babef462c554755a8b62fb79c59ebe5 2023-03-30 16:55:01,297 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/87a67fb20c2245f1ad933ec7b8bdbb47 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/87a67fb20c2245f1ad933ec7b8bdbb47 2023-03-30 16:55:01,297 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/cce8467d74d242a0863dd45913186f5a to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/cce8467d74d242a0863dd45913186f5a 2023-03-30 16:55:01,298 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/7457f10778524d9fb0f88e4fe996d4c7 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/7457f10778524d9fb0f88e4fe996d4c7 2023-03-30 16:55:01,299 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/7777c4885d3c473887073a86e17a4388 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/7777c4885d3c473887073a86e17a4388 2023-03-30 16:55:01,300 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/3be67ce6778d4a37934e3a5d195361bf to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/3be67ce6778d4a37934e3a5d195361bf 2023-03-30 16:55:01,301 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/d912324bc74b479ba8fa9d6bd2e5eb80 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/d912324bc74b479ba8fa9d6bd2e5eb80 2023-03-30 16:55:01,302 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/b0cf6884064e4d24a71e7938fc984a67 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/b0cf6884064e4d24a71e7938fc984a67 2023-03-30 16:55:01,302 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/a9205a1bb7fc4c8685bc0a73e3337cc7 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/a9205a1bb7fc4c8685bc0a73e3337cc7 2023-03-30 16:55:01,303 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/e79458c7b7254fa79f9f0e996bd3bbf6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/e79458c7b7254fa79f9f0e996bd3bbf6 2023-03-30 16:55:01,304 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/83ae7aa098ad4ac480a69dfaf732786f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/83ae7aa098ad4ac480a69dfaf732786f 2023-03-30 16:55:01,305 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/0746da17b0844507a331b3ff81e2ba2a to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/0746da17b0844507a331b3ff81e2ba2a 2023-03-30 16:55:01,305 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/d4a88e39c4ce46f996af350011242962 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/d4a88e39c4ce46f996af350011242962 2023-03-30 16:55:01,306 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/149f2accf2a1440d93186e1d0ee7dfe3 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/149f2accf2a1440d93186e1d0ee7dfe3 2023-03-30 16:55:01,307 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/8ebfd50998d1486d851ddad2458334c0 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/8ebfd50998d1486d851ddad2458334c0 2023-03-30 16:55:01,308 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/b0b6eceff6414c4a858c8b191f1f000c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/b0b6eceff6414c4a858c8b191f1f000c 2023-03-30 16:55:01,309 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/9b679f2a311b432ba1243bda12e1a03f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/9b679f2a311b432ba1243bda12e1a03f 2023-03-30 16:55:01,309 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/b4786236586b4fd8a55579b045fd4827 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/b4786236586b4fd8a55579b045fd4827 2023-03-30 16:55:01,310 DEBUG [StoreCloser-TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/ab1f393d420141b3b0885101836d3e23 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/ab1f393d420141b3b0885101836d3e23 2023-03-30 16:55:01,314 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/recovered.edits/644.seqid, newMaxSeqId=644, maxSeqId=1 2023-03-30 16:55:01,315 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c. 2023-03-30 16:55:01,315 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2720c59305e818889603edf8ccad1c2c: 2023-03-30 16:55:01,316 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2720c59305e818889603edf8ccad1c2c 2023-03-30 16:55:01,317 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=2720c59305e818889603edf8ccad1c2c, regionState=CLOSED 2023-03-30 16:55:01,317 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195301317"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680195301317"}]},"ts":"1680195301317"} 2023-03-30 16:55:01,319 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-03-30 16:55:01,319 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 2720c59305e818889603edf8ccad1c2c, server=jenkins-hbase4.apache.org,45193,1680195214285 in 1.3910 sec 2023-03-30 16:55:01,320 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-03-30 16:55:01,320 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2720c59305e818889603edf8ccad1c2c, UNASSIGN in 1.3940 sec 2023-03-30 16:55:01,321 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195301320"}]},"ts":"1680195301320"} 2023-03-30 16:55:01,321 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-30 16:55:01,323 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-30 16:55:01,324 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.4060 sec 2023-03-30 16:55:02,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-30 16:55:02,025 INFO [Listener at localhost/38075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-03-30 16:55:02,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-30 16:55:02,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:55:02,028 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:55:02,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-30 16:55:02,028 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:55:02,033 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c 2023-03-30 16:55:02,034 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A, FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B, FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C, FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/recovered.edits] 2023-03-30 16:55:02,037 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/7310488436d544cab20c92fcbfb05a92 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/7310488436d544cab20c92fcbfb05a92 2023-03-30 16:55:02,038 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/914d819760914bf78cf57f7c0c2b5009 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/914d819760914bf78cf57f7c0c2b5009 2023-03-30 16:55:02,039 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/af2a32a2e3e74971a37c0fedb7ce4566 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/A/af2a32a2e3e74971a37c0fedb7ce4566 2023-03-30 16:55:02,041 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/acd46be900bc47ed87649142265ebd8c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/acd46be900bc47ed87649142265ebd8c 2023-03-30 16:55:02,042 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/b5fca82aca314125af27344d7277bf51 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/b5fca82aca314125af27344d7277bf51 2023-03-30 16:55:02,043 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/d27bcf6a5e33494ea9f1c76ea2c0933b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/B/d27bcf6a5e33494ea9f1c76ea2c0933b 2023-03-30 16:55:02,045 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/3994808d145e4caeb4d3693a8acffba2 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/3994808d145e4caeb4d3693a8acffba2 2023-03-30 16:55:02,046 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/85b5311d5ec545aca4eebb4e5e1bc404 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/85b5311d5ec545aca4eebb4e5e1bc404 2023-03-30 16:55:02,047 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/966d7e4099184ada91f9feaefa99a6a1 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/966d7e4099184ada91f9feaefa99a6a1 2023-03-30 16:55:02,048 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/dc2f12d62fc040638fb0733de31f6749 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/C/dc2f12d62fc040638fb0733de31f6749 2023-03-30 16:55:02,051 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/recovered.edits/644.seqid to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c/recovered.edits/644.seqid 2023-03-30 16:55:02,051 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/2720c59305e818889603edf8ccad1c2c 2023-03-30 16:55:02,051 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 16:55:02,053 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:55:02,054 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-30 16:55:02,055 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-30 16:55:02,056 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:55:02,056 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-30 16:55:02,056 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680195302056"}]},"ts":"9223372036854775807"} 2023-03-30 16:55:02,057 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-30 16:55:02,058 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2720c59305e818889603edf8ccad1c2c, NAME => 'TestAcidGuarantees,,1680195276471.2720c59305e818889603edf8ccad1c2c.', STARTKEY => '', ENDKEY => ''}] 2023-03-30 16:55:02,058 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-30 16:55:02,058 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680195302058"}]},"ts":"9223372036854775807"} 2023-03-30 16:55:02,059 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-30 16:55:02,060 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:55:02,061 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 34 msec 2023-03-30 16:55:02,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-30 16:55:02,129 INFO [Listener at localhost/38075] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-03-30 16:55:02,141 INFO [Listener at localhost/38075] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=228 (was 228), OpenFileDescriptor=535 (was 555), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=416 (was 371) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=4514 (was 4635) 2023-03-30 16:55:02,152 INFO [Listener at localhost/38075] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=228, OpenFileDescriptor=535, MaxFileDescriptor=60000, SystemLoadAverage=416, ProcessCount=169, AvailableMemoryMB=4513 2023-03-30 16:55:02,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-30 16:55:02,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-30 16:55:02,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-30 16:55:02,156 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-30 16:55:02,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-03-30 16:55:02,157 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-30 16:55:02,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-30 16:55:02,159 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:02,159 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb empty. 2023-03-30 16:55:02,160 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:02,160 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 16:55:02,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-30 16:55:02,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-30 16:55:02,567 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-30 16:55:02,569 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => e3fe0ff55782e6f5d2f55c861f91d3cb, NAME => 'TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp 2023-03-30 16:55:02,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-30 16:55:02,974 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:55:02,974 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing e3fe0ff55782e6f5d2f55c861f91d3cb, disabling compactions & flushes 2023-03-30 16:55:02,974 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:02,975 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:02,975 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. after waiting 0 ms 2023-03-30 16:55:02,975 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:02,975 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:02,975 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:02,977 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-30 16:55:02,978 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195302977"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680195302977"}]},"ts":"1680195302977"} 2023-03-30 16:55:02,978 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-30 16:55:02,979 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-30 16:55:02,979 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195302979"}]},"ts":"1680195302979"} 2023-03-30 16:55:02,980 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-30 16:55:02,984 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e3fe0ff55782e6f5d2f55c861f91d3cb, ASSIGN}] 2023-03-30 16:55:02,985 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e3fe0ff55782e6f5d2f55c861f91d3cb, ASSIGN 2023-03-30 16:55:02,985 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e3fe0ff55782e6f5d2f55c861f91d3cb, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,45193,1680195214285; forceNewPlan=false, retain=false 2023-03-30 16:55:03,137 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=e3fe0ff55782e6f5d2f55c861f91d3cb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:03,137 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195303136"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680195303136"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680195303136"}]},"ts":"1680195303136"} 2023-03-30 16:55:03,138 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285}] 2023-03-30 16:55:03,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-30 16:55:03,292 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:03,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e3fe0ff55782e6f5d2f55c861f91d3cb, NAME => 'TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb.', STARTKEY => '', ENDKEY => ''} 2023-03-30 16:55:03,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:03,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:55:03,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:03,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:03,294 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:03,295 DEBUG [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A 2023-03-30 16:55:03,295 DEBUG [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A 2023-03-30 16:55:03,295 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:55:03,295 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e3fe0ff55782e6f5d2f55c861f91d3cb columnFamilyName A 2023-03-30 16:55:03,296 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] regionserver.HStore(310): Store=e3fe0ff55782e6f5d2f55c861f91d3cb/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:55:03,296 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:03,297 DEBUG [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B 2023-03-30 16:55:03,297 DEBUG [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B 2023-03-30 16:55:03,297 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:55:03,297 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e3fe0ff55782e6f5d2f55c861f91d3cb columnFamilyName B 2023-03-30 16:55:03,297 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] regionserver.HStore(310): Store=e3fe0ff55782e6f5d2f55c861f91d3cb/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:55:03,297 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:03,298 DEBUG [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C 2023-03-30 16:55:03,298 DEBUG [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C 2023-03-30 16:55:03,298 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:55:03,298 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e3fe0ff55782e6f5d2f55c861f91d3cb columnFamilyName C 2023-03-30 16:55:03,299 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] regionserver.HStore(310): Store=e3fe0ff55782e6f5d2f55c861f91d3cb/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:55:03,299 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:03,299 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:03,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:03,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 16:55:03,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:03,303 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-30 16:55:03,304 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e3fe0ff55782e6f5d2f55c861f91d3cb; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59234922, jitterRate=-0.1173308789730072}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 16:55:03,304 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:03,304 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb., pid=37, masterSystemTime=1680195303289 2023-03-30 16:55:03,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:03,305 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:03,305 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=e3fe0ff55782e6f5d2f55c861f91d3cb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:03,306 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195303305"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680195303305"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680195303305"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680195303305"}]},"ts":"1680195303305"} 2023-03-30 16:55:03,307 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-03-30 16:55:03,307 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 in 168 msec 2023-03-30 16:55:03,308 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-03-30 16:55:03,308 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e3fe0ff55782e6f5d2f55c861f91d3cb, ASSIGN in 323 msec 2023-03-30 16:55:03,309 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-30 16:55:03,309 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195303309"}]},"ts":"1680195303309"} 2023-03-30 16:55:03,309 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-30 16:55:03,311 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-30 16:55:03,312 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1570 sec 2023-03-30 16:55:04,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-30 16:55:04,262 INFO [Listener at localhost/38075] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-03-30 16:55:04,262 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x6038da2f to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:04,267 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22d19185, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:04,268 DEBUG [Listener at localhost/38075] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:04,270 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60862, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:04,271 DEBUG [Listener at localhost/38075] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-30 16:55:04,272 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60548, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-30 16:55:04,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-30 16:55:04,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-30 16:55:04,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-30 16:55:04,687 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-30 16:55:04,688 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-30 16:55:04,688 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-30 16:55:04,690 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-30 16:55:04,693 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e3fe0ff55782e6f5d2f55c861f91d3cb, REOPEN/MOVE}] 2023-03-30 16:55:04,693 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e3fe0ff55782e6f5d2f55c861f91d3cb, REOPEN/MOVE 2023-03-30 16:55:04,694 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=e3fe0ff55782e6f5d2f55c861f91d3cb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:04,694 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195304694"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680195304694"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680195304694"}]},"ts":"1680195304694"} 2023-03-30 16:55:04,695 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285}] 2023-03-30 16:55:04,847 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:04,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e3fe0ff55782e6f5d2f55c861f91d3cb, disabling compactions & flushes 2023-03-30 16:55:04,848 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:04,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:04,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. after waiting 0 ms 2023-03-30 16:55:04,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:04,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-30 16:55:04,851 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:04,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:04,851 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: e3fe0ff55782e6f5d2f55c861f91d3cb to self. 2023-03-30 16:55:04,852 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:04,853 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=e3fe0ff55782e6f5d2f55c861f91d3cb, regionState=CLOSED 2023-03-30 16:55:04,853 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195304853"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680195304853"}]},"ts":"1680195304853"} 2023-03-30 16:55:04,855 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-03-30 16:55:04,855 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 in 159 msec 2023-03-30 16:55:04,855 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e3fe0ff55782e6f5d2f55c861f91d3cb, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,45193,1680195214285; forceNewPlan=false, retain=true 2023-03-30 16:55:05,006 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=e3fe0ff55782e6f5d2f55c861f91d3cb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,006 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195305005"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680195305005"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680195305005"}]},"ts":"1680195305005"} 2023-03-30 16:55:05,007 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285}] 2023-03-30 16:55:05,161 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:05,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e3fe0ff55782e6f5d2f55c861f91d3cb, NAME => 'TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb.', STARTKEY => '', ENDKEY => ''} 2023-03-30 16:55:05,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:05,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:55:05,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:05,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:05,163 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:05,164 DEBUG [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A 2023-03-30 16:55:05,164 DEBUG [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A 2023-03-30 16:55:05,164 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:55:05,164 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e3fe0ff55782e6f5d2f55c861f91d3cb columnFamilyName A 2023-03-30 16:55:05,165 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] regionserver.HStore(310): Store=e3fe0ff55782e6f5d2f55c861f91d3cb/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:55:05,166 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:05,166 DEBUG [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B 2023-03-30 16:55:05,166 DEBUG [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B 2023-03-30 16:55:05,167 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:55:05,167 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e3fe0ff55782e6f5d2f55c861f91d3cb columnFamilyName B 2023-03-30 16:55:05,167 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] regionserver.HStore(310): Store=e3fe0ff55782e6f5d2f55c861f91d3cb/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:55:05,167 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:05,168 DEBUG [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C 2023-03-30 16:55:05,168 DEBUG [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C 2023-03-30 16:55:05,168 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:55:05,168 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e3fe0ff55782e6f5d2f55c861f91d3cb columnFamilyName C 2023-03-30 16:55:05,168 INFO [StoreOpener-e3fe0ff55782e6f5d2f55c861f91d3cb-1] regionserver.HStore(310): Store=e3fe0ff55782e6f5d2f55c861f91d3cb/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:55:05,169 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:05,169 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:05,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:05,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 16:55:05,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:05,173 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e3fe0ff55782e6f5d2f55c861f91d3cb; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75030640, jitterRate=0.11804366111755371}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 16:55:05,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:05,174 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb., pid=42, masterSystemTime=1680195305158 2023-03-30 16:55:05,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:05,175 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:05,175 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=e3fe0ff55782e6f5d2f55c861f91d3cb, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,175 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195305175"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680195305175"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680195305175"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680195305175"}]},"ts":"1680195305175"} 2023-03-30 16:55:05,177 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-03-30 16:55:05,177 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 in 169 msec 2023-03-30 16:55:05,178 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-03-30 16:55:05,178 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e3fe0ff55782e6f5d2f55c861f91d3cb, REOPEN/MOVE in 484 msec 2023-03-30 16:55:05,180 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-03-30 16:55:05,180 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 489 msec 2023-03-30 16:55:05,181 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 906 msec 2023-03-30 16:55:05,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-03-30 16:55:05,183 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x41798291 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:05,187 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67b6d133, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:05,187 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x118e131f to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:05,190 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1cee66b5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:05,191 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x2deaf1fa to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:05,195 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c1d19b3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:05,196 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x6e7e18ef to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:05,199 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14e151fb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:05,199 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x5b7d4a51 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:05,203 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@455a88b1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:05,203 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x722b2407 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:05,206 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63a535f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:05,206 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x066bf91c to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:05,223 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d4bfeca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:05,224 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x6846dddb to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:05,229 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@313c7ad6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:05,229 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x7ea43962 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:05,232 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20d28b68, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:05,233 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x2abd7961 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:05,236 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@750d2a7d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:05,249 DEBUG [hconnection-0x1e12b30e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:05,249 DEBUG [hconnection-0x593c0946-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:05,249 DEBUG [hconnection-0x79fccfab-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:05,249 DEBUG [hconnection-0x6b29b7fd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:05,250 DEBUG [hconnection-0x62a15841-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:05,250 DEBUG [hconnection-0x63d36782-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:05,250 DEBUG [hconnection-0x784d4558-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:05,253 DEBUG [hconnection-0x5b43715b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:05,253 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60866, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:05,253 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60880, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:05,254 DEBUG [hconnection-0x13188802-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:05,254 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60884, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:05,254 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60892, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:05,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:05,254 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60864, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:05,256 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60882, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:05,257 DEBUG [hconnection-0x5ed4d14c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:05,257 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60908, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:05,259 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60938, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:05,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:05,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:05,261 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:05,261 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:05,261 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60910, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:05,262 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:05,262 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:05,263 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60926, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:05,263 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,263 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,266 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:05,266 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:05,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:05,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:05,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:05,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,268 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:05,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:05,268 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:05,268 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:05,268 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:05,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:05,270 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:55:05,271 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:05,272 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:05,272 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:05,272 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:05,272 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:05,272 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:05,276 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:05,276 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:05,276 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:05,276 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:05,276 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:05,277 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:05,277 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:05,277 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:05,277 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,277 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:05,277 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:05,278 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:05,284 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:05,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:05,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:05,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:05,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bbc4955[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:05,287 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:05,287 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:05,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195365284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,288 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,288 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:05,288 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:05,289 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:05,289 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,289 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:05,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:05,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195365286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:05,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195365287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:05,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195365288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:05,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195365288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,294 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,294 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:05,294 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:05,295 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:05,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:05,297 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:05,297 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:05,297 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:05,297 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:05,297 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:05,298 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,299 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:05,299 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:05,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:05,299 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:05,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,300 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:05,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,302 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,302 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,302 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:05,302 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:05,302 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,302 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,302 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,302 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:05,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:05,302 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:05,302 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:05,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:05,302 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:05,303 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:05,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:05,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:05,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:05,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:05,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195365389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:05,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195365391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:05,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195365391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:05,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195365391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:05,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195365391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,403 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:05,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:05,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:05,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:05,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:05,417 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:05,417 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:05,417 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:05,417 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:05,417 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,418 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,419 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:05,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:05,419 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,419 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:05,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:05,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:05,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:05,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:05,423 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:05,423 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:05,423 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:05,424 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:05,424 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:05,424 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:05,424 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:05,424 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:05,424 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:05,424 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:05,424 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:05,424 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,424 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:05,424 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:05,427 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,427 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:05,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:05,427 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,427 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:05,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:05,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:05,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:05,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:05,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@159d44a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:05,429 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:05,429 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,430 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,430 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:05,430 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:05,430 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:05,430 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,430 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:05,431 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,431 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:05,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:05,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:05,432 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:05,432 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:05,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:05,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:05,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:05,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:05,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:05,435 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:05,435 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:05,435 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:05,435 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:05,435 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:05,437 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,437 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:05,437 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:05,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:05,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:05,438 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:05,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:05,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:05,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,439 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:05,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,442 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,442 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:05,442 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,442 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:05,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:05,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:05,442 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:05,442 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:05,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:05,443 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:05,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:05,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:05,443 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:05,443 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,443 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,443 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,443 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:05,544 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:05,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:05,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:05,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:05,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:05,556 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:05,556 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:05,556 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:05,556 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:05,557 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,557 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,558 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,558 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:05,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:05,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:05,559 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,559 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:05,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:05,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:05,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:05,564 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:05,564 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:05,564 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:05,567 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:05,567 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:05,568 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:05,568 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:05,568 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:05,568 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:05,568 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:05,568 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:05,568 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,568 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:05,568 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:05,570 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:05,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:05,571 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,571 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:05,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:05,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e5fbb01[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:05,573 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:05,573 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,575 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,576 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:05,576 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,576 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:05,576 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:05,576 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:05,579 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:05,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:05,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:05,580 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:05,580 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:05,580 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:05,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:05,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:05,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:05,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:05,584 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:05,584 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:05,584 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:05,584 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:05,584 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:05,586 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:05,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:05,586 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,586 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:05,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:05,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,587 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,589 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:05,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,592 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:05,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:05,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:05,592 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:05,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:05,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:05,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:05,592 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:05,592 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:05,592 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:05,592 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,592 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,592 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,593 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,593 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,593 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,593 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:05,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:05,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:05,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195365594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:05,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195365594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:05,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195365595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:05,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195365595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:05,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195365596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,692 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:05,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:05,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:05,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:05,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:05,696 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:05,696 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:05,696 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:05,696 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:05,696 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,696 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,698 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:05,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:05,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:05,698 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,698 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,698 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:05,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:05,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:05,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:05,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:05,699 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:05,699 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:05,699 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:05,699 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:05,699 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:05,700 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:05,700 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:05,700 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:05,700 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:05,700 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:05,700 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:05,700 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,700 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:05,700 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:05,701 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330cfa61fe2cc784940a81e50d0beb49243 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330cfa61fe2cc784940a81e50d0beb49243 2023-03-30 16:55:05,703 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,703 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:05,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:05,704 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,704 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:05,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:05,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12b7ee33[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:05,705 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:05,705 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,706 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,707 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:05,707 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:05,707 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:05,707 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,707 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:05,708 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:05,708 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:05,708 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:05,712 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:05,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:05,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:05,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:05,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:05,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:05,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:05,714 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:05,714 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:05,714 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:05,715 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:05,715 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:05,716 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,716 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:05,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:05,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:05,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,717 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:05,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:05,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:05,718 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,719 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/c202c30c81c44d6cb028459aae08eefe 2023-03-30 16:55:05,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:05,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,720 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:05,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,727 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:05,727 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:05,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:05,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:05,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:05,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:05,728 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:05,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:05,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:05,728 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:05,728 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:05,728 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,728 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,728 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,728 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,728 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,728 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,728 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:05,760 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/b8c04902a5ef44cba7ac0df1a9618b18 2023-03-30 16:55:05,828 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:05,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:05,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:05,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:05,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:05,836 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:05,836 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:05,836 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:05,836 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:05,837 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,837 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,838 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,838 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:05,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:05,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:05,838 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,838 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:05,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:05,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:05,841 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:05,841 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:05,841 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:05,842 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:05,842 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:05,842 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:05,842 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:05,842 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:05,842 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:05,843 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:05,843 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:05,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:05,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:05,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:05,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:05,844 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,844 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:05,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:05,845 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,845 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:05,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:05,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:05,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:05,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:05,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bb8fbd1[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:05,848 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:05,848 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,849 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,849 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:05,849 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:05,849 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:05,849 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,849 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:05,851 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,851 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:05,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:05,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:05,851 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:05,851 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:05,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:05,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:05,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:05,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:05,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:05,854 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:05,854 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:05,854 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:05,854 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:05,854 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:05,855 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,855 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,855 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:05,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:05,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:05,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:05,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:05,857 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,858 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,860 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,860 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,860 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,860 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:05,861 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:05,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:05,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:05,861 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,860 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:05,861 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:05,861 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:05,861 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:05,861 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:05,861 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:05,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:05,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195365899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:05,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195365899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:05,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195365901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:05,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195365903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:05,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195365905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,961 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:05,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:05,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:05,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:05,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:05,964 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:05,964 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:05,965 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:05,965 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:05,965 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,965 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,966 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:05,966 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:05,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:05,966 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,966 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:05,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:05,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:05,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:05,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:05,967 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:05,968 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:05,968 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:05,968 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:05,968 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:05,968 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:05,968 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:05,968 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:05,968 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:05,968 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:05,968 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:05,968 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,968 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:05,968 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:05,971 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:05,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:05,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:05,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:05,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@248c9968[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:05,972 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:05,972 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,973 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,973 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:05,973 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:05,974 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:05,974 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,974 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:05,975 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:05,975 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:05,975 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,975 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:05,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:05,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:05,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:05,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:05,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:05,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:05,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:05,977 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:05,977 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:05,977 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:05,977 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:05,977 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:05,979 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,979 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:05,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,979 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:05,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:05,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:05,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:05,980 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:05,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:05,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,981 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:05,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:05,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,989 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:05,989 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:05,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:05,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:05,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:05,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:05,989 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,989 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,989 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:05,989 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:05,989 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,989 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:05,989 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:05,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:05,989 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:05,989 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:05,989 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:05,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:05,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:06,089 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:06,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:06,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:06,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:06,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:06,093 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:06,093 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:06,093 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:06,093 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:06,093 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,093 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:06,094 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:06,094 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:06,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:06,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:06,095 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,095 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:06,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:06,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:06,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:06,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:06,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:06,096 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:06,096 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:06,096 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:06,096 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:06,096 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:06,096 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:06,096 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:06,096 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:06,096 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:06,096 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:06,096 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:06,096 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,096 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:06,096 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:06,098 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,098 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:06,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:06,098 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,098 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:06,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:06,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:06,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:06,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:06,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:06,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47e8a94b[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:06,099 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:06,100 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,101 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,101 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:06,101 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:06,101 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:06,101 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,101 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:06,103 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,103 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:06,103 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:06,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:06,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:06,103 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:06,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:06,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:06,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:06,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:06,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:06,106 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:06,106 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:06,106 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:06,106 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:06,106 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:06,107 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,107 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:06,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:06,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,107 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:06,108 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:06,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:06,109 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:06,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:06,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,115 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,115 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:06,115 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:06,115 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:06,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:06,115 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:06,115 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:06,115 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:06,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:06,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:06,115 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,115 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:06,115 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,115 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,115 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:06,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:06,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:06,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:06,188 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/3cf07530b0f942e687b4053caf9cb10a 2023-03-30 16:55:06,193 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/c202c30c81c44d6cb028459aae08eefe as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c202c30c81c44d6cb028459aae08eefe 2023-03-30 16:55:06,196 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c202c30c81c44d6cb028459aae08eefe, entries=150, sequenceid=17, filesize=25.1 K 2023-03-30 16:55:06,197 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/b8c04902a5ef44cba7ac0df1a9618b18 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/b8c04902a5ef44cba7ac0df1a9618b18 2023-03-30 16:55:06,201 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/b8c04902a5ef44cba7ac0df1a9618b18, entries=150, sequenceid=17, filesize=11.6 K 2023-03-30 16:55:06,203 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/3cf07530b0f942e687b4053caf9cb10a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/3cf07530b0f942e687b4053caf9cb10a 2023-03-30 16:55:06,207 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/3cf07530b0f942e687b4053caf9cb10a, entries=150, sequenceid=17, filesize=11.6 K 2023-03-30 16:55:06,207 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for e3fe0ff55782e6f5d2f55c861f91d3cb in 937ms, sequenceid=17, compaction requested=false 2023-03-30 16:55:06,207 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:06,215 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:06,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:06,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:06,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:06,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:06,218 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:06,218 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:06,218 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:06,218 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:06,219 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,219 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:06,224 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:06,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:06,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:06,224 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:06,225 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,225 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:06,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:06,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:06,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:06,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:06,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:06,225 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:06,225 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:06,225 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:06,226 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:06,226 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:06,226 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:06,226 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:06,226 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-30 16:55:06,226 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:06,226 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:06,226 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:06,226 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:06,226 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:06,226 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:06,242 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330b65585e825d34ce488e7eebd4b26cb64 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330b65585e825d34ce488e7eebd4b26cb64 2023-03-30 16:55:06,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:06,404 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:06,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:06,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195366418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:06,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195366418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:06,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195366419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:06,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195366422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:06,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195366426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:06,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195366526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:06,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195366526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:06,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195366526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:06,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195366530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:06,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195366530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,648 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/406f8e923ce14c1d847d4ca94e6b3f40 2023-03-30 16:55:06,663 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/59d515e3b3fe4b8dba458d99e55e1dde 2023-03-30 16:55:06,675 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/0c129f8304554bf0a18e1726c4a68640 2023-03-30 16:55:06,679 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/406f8e923ce14c1d847d4ca94e6b3f40 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/406f8e923ce14c1d847d4ca94e6b3f40 2023-03-30 16:55:06,685 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/406f8e923ce14c1d847d4ca94e6b3f40, entries=150, sequenceid=36, filesize=25.1 K 2023-03-30 16:55:06,686 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/59d515e3b3fe4b8dba458d99e55e1dde as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/59d515e3b3fe4b8dba458d99e55e1dde 2023-03-30 16:55:06,692 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/59d515e3b3fe4b8dba458d99e55e1dde, entries=150, sequenceid=36, filesize=11.6 K 2023-03-30 16:55:06,692 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/0c129f8304554bf0a18e1726c4a68640 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/0c129f8304554bf0a18e1726c4a68640 2023-03-30 16:55:06,697 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/0c129f8304554bf0a18e1726c4a68640, entries=150, sequenceid=36, filesize=11.6 K 2023-03-30 16:55:06,698 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for e3fe0ff55782e6f5d2f55c861f91d3cb in 472ms, sequenceid=36, compaction requested=false 2023-03-30 16:55:06,698 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:06,698 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:06,698 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:06,698 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:06,698 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,698 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:06,698 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:06,701 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:06,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:06,702 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:06,702 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,702 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:06,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:06,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:06,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:06,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a8301f3[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:06,703 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:06,703 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,704 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:06,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:06,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:06,705 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,705 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:06,706 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,706 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:06,706 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:06,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:06,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:06,706 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:06,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:06,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:06,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:06,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:06,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:06,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:06,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:06,709 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:06,709 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:06,709 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:06,709 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:06,709 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:06,713 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,713 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:06,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:06,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:06,714 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,715 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,716 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:06,717 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:06,717 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:06,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:06,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:06,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:06,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:06,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:06,718 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,719 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:06,719 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:06,719 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,719 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:06,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:06,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:06,719 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,719 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,719 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:06,719 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:06,720 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:06,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:06,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:06,735 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-30 16:55:06,735 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:06,736 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:06,736 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:06,736 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:06,736 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:06,736 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:06,773 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033070de98dd096746aa9f0fc89032f67dc9 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033070de98dd096746aa9f0fc89032f67dc9 2023-03-30 16:55:06,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:06,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195366764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:06,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195366765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:06,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195366766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:06,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195366769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:06,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195366769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,786 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/45213b0b274d465091ddc7c78f8f598f 2023-03-30 16:55:06,806 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/782ac22386f44f83801710e76aa140e7 2023-03-30 16:55:06,820 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:06,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:06,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:06,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:06,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:06,832 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:06,832 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:06,832 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:06,832 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:06,832 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,832 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:06,834 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:06,834 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:06,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:06,834 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,834 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:06,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:06,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:06,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:06,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:06,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:06,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:06,837 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:06,837 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:06,837 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:06,840 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:06,840 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:06,840 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:06,840 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:06,840 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:06,840 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:06,840 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:06,840 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:06,840 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,840 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:06,840 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:06,842 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,842 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:06,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:06,842 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,842 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:06,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:06,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:06,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:06,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:06,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:06,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46c91901[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:06,844 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:06,844 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,846 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,846 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:06,846 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,846 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:06,846 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:06,846 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:06,847 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,847 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:06,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:06,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:06,847 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:06,847 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:06,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:06,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:06,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:06,849 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-30 16:55:06,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:06,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:06,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:06,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:06,851 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:06,851 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:06,851 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:06,851 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:06,851 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:06,853 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:06,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:06,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:06,854 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:06,854 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,854 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:06,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:06,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:06,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:06,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:06,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,856 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:06,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:06,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,860 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:06,860 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:06,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:06,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:06,868 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,868 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:06,868 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:06,868 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,868 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,868 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,868 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,868 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:06,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:06,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:06,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:06,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:06,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:06,869 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:06,869 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:06,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:06,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195366878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:06,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195366879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:06,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195366879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:06,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195366879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:06,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195366879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,969 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:06,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:06,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:06,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:06,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:06,972 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:06,972 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:06,972 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:06,972 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:06,972 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,972 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:06,975 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:06,975 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:06,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:06,976 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,976 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:06,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:06,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:06,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:06,977 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:06,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:06,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:06,977 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:06,977 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:06,977 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:06,977 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:06,977 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:06,977 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:06,977 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:06,977 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:06,977 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:06,977 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:06,977 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:06,977 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,977 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:06,977 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:06,979 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,979 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:06,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:06,979 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,979 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:06,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:06,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:06,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:06,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:06,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:06,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ce8f69[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:06,981 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:06,981 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,982 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,982 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:06,982 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:06,982 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:06,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:06,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:06,984 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:06,984 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:06,984 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:06,984 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:06,984 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:06,984 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:06,984 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:06,984 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:06,990 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,990 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,990 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:06,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:06,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:06,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:06,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:06,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:06,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,993 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:06,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:06,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:06,993 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:06,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:06,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,996 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:06,996 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:06,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:06,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:06,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:06,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:06,996 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,996 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:06,996 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:06,996 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:06,996 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,996 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:06,996 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:06,996 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:06,996 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:06,996 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:06,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:06,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:06,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:07,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:07,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195367087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:07,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:07,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195367087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195367088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:07,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195367088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:07,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195367089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,096 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:07,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:07,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:07,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:07,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:07,099 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:07,099 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:07,099 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:07,099 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:07,100 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,100 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:07,103 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:07,103 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:07,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:07,104 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:07,104 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:07,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:07,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:07,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:07,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:07,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:07,105 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:07,105 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:07,105 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:07,105 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:07,105 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:07,105 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:07,105 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:07,105 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:07,105 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:07,105 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:07,105 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:07,105 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,105 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:07,105 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:07,107 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,108 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:07,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:07,108 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,108 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:07,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:07,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:07,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:07,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:07,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:07,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eedc0a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:07,111 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:07,111 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,112 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,112 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:07,112 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:07,112 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,112 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:07,112 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:07,113 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:07,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:07,113 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:07,113 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:07,113 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:07,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:07,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:07,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:07,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:07,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:07,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:07,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:07,115 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:07,115 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:07,115 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:07,115 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:07,115 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:07,117 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,117 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,117 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:07,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:07,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:07,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:07,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:07,118 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:07,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:07,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:07,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:07,118 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:07,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:07,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,121 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,121 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:07,121 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:07,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:07,121 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:07,121 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:07,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:07,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:07,121 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:07,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:07,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:07,121 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,122 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,122 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,122 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,122 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:07,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:07,122 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:07,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:07,222 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:07,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:07,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:07,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:07,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:07,225 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:07,226 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:07,226 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:07,226 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:07,226 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,226 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:07,227 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:07,227 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:07,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:07,227 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,227 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:07,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:07,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:07,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:07,228 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/6f50d38c154e470b9e8fdfde9db958d3 2023-03-30 16:55:07,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:07,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:07,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:07,229 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:07,229 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:07,229 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:07,229 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:07,229 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:07,229 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:07,229 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:07,229 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:07,229 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:07,229 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:07,229 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:07,229 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,229 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:07,230 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:07,231 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:07,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:07,231 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:07,232 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,232 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:07,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:07,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:07,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:07,233 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/45213b0b274d465091ddc7c78f8f598f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/45213b0b274d465091ddc7c78f8f598f 2023-03-30 16:55:07,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:07,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7610d9fd[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:07,233 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:07,233 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,234 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,234 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:07,234 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:07,234 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:07,234 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,235 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:07,237 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:07,237 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:07,237 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:07,237 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/45213b0b274d465091ddc7c78f8f598f, entries=200, sequenceid=50, filesize=31.9 K 2023-03-30 16:55:07,237 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:07,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:07,238 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/782ac22386f44f83801710e76aa140e7 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/782ac22386f44f83801710e76aa140e7 2023-03-30 16:55:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:07,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:07,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:07,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:07,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:07,240 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:07,240 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:07,240 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:07,240 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:07,240 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:07,241 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/782ac22386f44f83801710e76aa140e7, entries=150, sequenceid=50, filesize=11.6 K 2023-03-30 16:55:07,242 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,242 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:07,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:07,242 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:07,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:07,242 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/6f50d38c154e470b9e8fdfde9db958d3 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/6f50d38c154e470b9e8fdfde9db958d3 2023-03-30 16:55:07,242 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:07,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:07,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:07,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:07,243 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:07,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:07,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:07,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,246 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/6f50d38c154e470b9e8fdfde9db958d3, entries=150, sequenceid=50, filesize=11.6 K 2023-03-30 16:55:07,246 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:07,246 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:07,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:07,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:07,246 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,246 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:07,247 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:07,247 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:07,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:07,247 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for e3fe0ff55782e6f5d2f55c861f91d3cb in 512ms, sequenceid=50, compaction requested=true 2023-03-30 16:55:07,247 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:07,247 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:07,247 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:55:07,247 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:55:07,247 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:07,247 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:07,248 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,248 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,248 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:07,248 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:07,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:07,248 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:55:07,248 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): e3fe0ff55782e6f5d2f55c861f91d3cb/A is initiating minor compaction (all files) 2023-03-30 16:55:07,248 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:07,248 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:07,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:07,248 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e3fe0ff55782e6f5d2f55c861f91d3cb/A in TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:07,249 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:55:07,249 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c202c30c81c44d6cb028459aae08eefe, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/406f8e923ce14c1d847d4ca94e6b3f40, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/45213b0b274d465091ddc7c78f8f598f] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp, totalSize=82.2 K 2023-03-30 16:55:07,249 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): e3fe0ff55782e6f5d2f55c861f91d3cb/B is initiating minor compaction (all files) 2023-03-30 16:55:07,249 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3fe0ff55782e6f5d2f55c861f91d3cb/B in TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:07,249 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/b8c04902a5ef44cba7ac0df1a9618b18, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/59d515e3b3fe4b8dba458d99e55e1dde, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/782ac22386f44f83801710e76aa140e7] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp, totalSize=34.8 K 2023-03-30 16:55:07,249 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting c202c30c81c44d6cb028459aae08eefe, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680195305264 2023-03-30 16:55:07,249 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting b8c04902a5ef44cba7ac0df1a9618b18, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680195305264 2023-03-30 16:55:07,250 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 406f8e923ce14c1d847d4ca94e6b3f40, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680195305281 2023-03-30 16:55:07,250 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 59d515e3b3fe4b8dba458d99e55e1dde, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680195305281 2023-03-30 16:55:07,250 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 45213b0b274d465091ddc7c78f8f598f, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680195306416 2023-03-30 16:55:07,250 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 782ac22386f44f83801710e76aa140e7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680195306416 2023-03-30 16:55:07,265 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): e3fe0ff55782e6f5d2f55c861f91d3cb#B#compaction#281 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:07,281 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): e3fe0ff55782e6f5d2f55c861f91d3cb#A#compaction#282 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:07,288 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/e8bbd703c8fc408ab94f92d82cfdb81f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/e8bbd703c8fc408ab94f92d82cfdb81f 2023-03-30 16:55:07,294 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3fe0ff55782e6f5d2f55c861f91d3cb/B of e3fe0ff55782e6f5d2f55c861f91d3cb into e8bbd703c8fc408ab94f92d82cfdb81f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:07,294 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:07,294 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb., storeName=e3fe0ff55782e6f5d2f55c861f91d3cb/B, priority=13, startTime=1680195307247; duration=0sec 2023-03-30 16:55:07,294 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:07,294 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:55:07,295 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:55:07,295 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): e3fe0ff55782e6f5d2f55c861f91d3cb/C is initiating minor compaction (all files) 2023-03-30 16:55:07,295 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3fe0ff55782e6f5d2f55c861f91d3cb/C in TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:07,295 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/3cf07530b0f942e687b4053caf9cb10a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/0c129f8304554bf0a18e1726c4a68640, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/6f50d38c154e470b9e8fdfde9db958d3] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp, totalSize=34.8 K 2023-03-30 16:55:07,297 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 3cf07530b0f942e687b4053caf9cb10a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1680195305264 2023-03-30 16:55:07,297 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 0c129f8304554bf0a18e1726c4a68640, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680195305281 2023-03-30 16:55:07,298 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 6f50d38c154e470b9e8fdfde9db958d3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680195306416 2023-03-30 16:55:07,303 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): e3fe0ff55782e6f5d2f55c861f91d3cb#C#compaction#283 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:07,314 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/6bb362391f05459babc371369e211283 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/6bb362391f05459babc371369e211283 2023-03-30 16:55:07,318 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3fe0ff55782e6f5d2f55c861f91d3cb/C of e3fe0ff55782e6f5d2f55c861f91d3cb into 6bb362391f05459babc371369e211283(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:07,318 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:07,318 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb., storeName=e3fe0ff55782e6f5d2f55c861f91d3cb/C, priority=13, startTime=1680195307247; duration=0sec 2023-03-30 16:55:07,318 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:07,348 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:07,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:07,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:07,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:07,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:07,352 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:07,352 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:07,352 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:07,352 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:07,352 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,352 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:07,353 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,353 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:07,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:07,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:07,354 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,354 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:07,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:07,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:07,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:07,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:07,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:07,355 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:07,355 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:07,355 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:07,355 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:07,355 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:07,355 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:07,355 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:07,355 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-30 16:55:07,355 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:07,355 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:07,355 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:07,355 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:07,355 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:07,355 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:07,368 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303307144daa2e7594e7ab2692db497c3675e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303307144daa2e7594e7ab2692db497c3675e 2023-03-30 16:55:07,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:07,397 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:07,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:07,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195367408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:07,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195367409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,415 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:07,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:07,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195367410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:07,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195367410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195367409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:07,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195367511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:07,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195367515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:07,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195367516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:07,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195367517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:07,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195367518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:07,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195367717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:07,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195367721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:07,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195367722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:07,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195367722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:07,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195367724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,786 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=70, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/73564254ccd94243b88f67dcd35228fb 2023-03-30 16:55:07,796 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/3b301ea7019a4d499efda9139729978f 2023-03-30 16:55:07,806 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/25aa9583403e4a35a21fddac5cdd89b3 2023-03-30 16:55:07,811 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/73564254ccd94243b88f67dcd35228fb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/73564254ccd94243b88f67dcd35228fb 2023-03-30 16:55:07,815 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/73564254ccd94243b88f67dcd35228fb, entries=150, sequenceid=70, filesize=25.1 K 2023-03-30 16:55:07,816 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/3b301ea7019a4d499efda9139729978f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/3b301ea7019a4d499efda9139729978f 2023-03-30 16:55:07,819 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/3b301ea7019a4d499efda9139729978f, entries=150, sequenceid=70, filesize=11.6 K 2023-03-30 16:55:07,820 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/25aa9583403e4a35a21fddac5cdd89b3 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/25aa9583403e4a35a21fddac5cdd89b3 2023-03-30 16:55:07,824 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/25aa9583403e4a35a21fddac5cdd89b3, entries=150, sequenceid=70, filesize=11.6 K 2023-03-30 16:55:07,825 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for e3fe0ff55782e6f5d2f55c861f91d3cb in 470ms, sequenceid=70, compaction requested=false 2023-03-30 16:55:07,825 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:07,825 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:07,825 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:07,825 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:07,825 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,828 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:07,828 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:07,830 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,830 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:07,831 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,831 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:07,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:07,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:07,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:07,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:07,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@781fd72f[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:07,833 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:07,833 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,834 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,834 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:07,834 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:07,834 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:07,834 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,834 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:07,836 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:07,836 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:07,836 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,836 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:07,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:07,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:07,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:07,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:07,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:07,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:07,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:07,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:07,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:07,839 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:07,839 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:07,839 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:07,839 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:07,839 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:07,841 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,841 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,841 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:07,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:07,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:07,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,842 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,843 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:07,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:07,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:07,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:07,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:07,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:07,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,847 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:07,847 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:07,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:07,847 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:07,847 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:07,847 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:07,847 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:07,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:07,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:07,847 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,848 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:07,848 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,847 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:07,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:07,848 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:07,848 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:07,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:07,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:07,948 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:07,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:07,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:07,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:07,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:07,951 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:07,951 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:07,951 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:07,951 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:07,951 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,951 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:07,952 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:07,952 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:07,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:07,953 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:07,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:07,953 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:07,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:07,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:07,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:07,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:07,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:07,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:07,954 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:07,954 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:07,954 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:07,955 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:07,955 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:07,955 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:07,955 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:07,955 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-30 16:55:07,955 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:07,955 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:07,955 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:07,955 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:07,955 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:07,955 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:07,972 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330085da5dce2f540bcb31076226aedfa1d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330085da5dce2f540bcb31076226aedfa1d 2023-03-30 16:55:08,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:08,026 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:08,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:08,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195368045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:08,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:08,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195368045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:08,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:08,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195368046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:08,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:08,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195368046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:08,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:08,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195368050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:08,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:08,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195368157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:08,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:08,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195368157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:08,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:08,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195368158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:08,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:08,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195368158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:08,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:08,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195368158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:08,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:08,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195368360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:08,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:08,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195368363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:08,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:08,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195368363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:08,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:08,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195368364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:08,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:08,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195368365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:08,378 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=84, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/6cf862c234024c48b523fe502089e003 2023-03-30 16:55:08,505 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/f3e258bc59f64f31b9c707026b0392ff as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/f3e258bc59f64f31b9c707026b0392ff 2023-03-30 16:55:08,509 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3fe0ff55782e6f5d2f55c861f91d3cb/A of e3fe0ff55782e6f5d2f55c861f91d3cb into f3e258bc59f64f31b9c707026b0392ff(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-30 16:55:08,509 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:08,509 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb., storeName=e3fe0ff55782e6f5d2f55c861f91d3cb/A, priority=13, startTime=1680195307247; duration=1sec 2023-03-30 16:55:08,509 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:08,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:08,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195368666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:08,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:08,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195368674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:08,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:08,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195368675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:08,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:08,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195368676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:08,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:08,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195368676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:08,802 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/c5d4d7e365b34a36963e766783856006 2023-03-30 16:55:09,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:09,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195369172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:09,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195369180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:09,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195369183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:09,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195369184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:09,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195369184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,215 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/a22a061eee32485d9cd4341fbb8444d5 2023-03-30 16:55:09,219 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/6cf862c234024c48b523fe502089e003 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/6cf862c234024c48b523fe502089e003 2023-03-30 16:55:09,222 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/6cf862c234024c48b523fe502089e003, entries=150, sequenceid=84, filesize=25.1 K 2023-03-30 16:55:09,223 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/c5d4d7e365b34a36963e766783856006 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/c5d4d7e365b34a36963e766783856006 2023-03-30 16:55:09,227 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/c5d4d7e365b34a36963e766783856006, entries=150, sequenceid=84, filesize=11.6 K 2023-03-30 16:55:09,230 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/a22a061eee32485d9cd4341fbb8444d5 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/a22a061eee32485d9cd4341fbb8444d5 2023-03-30 16:55:09,234 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/a22a061eee32485d9cd4341fbb8444d5, entries=150, sequenceid=84, filesize=11.6 K 2023-03-30 16:55:09,234 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for e3fe0ff55782e6f5d2f55c861f91d3cb in 1279ms, sequenceid=84, compaction requested=true 2023-03-30 16:55:09,235 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:09,235 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:09,235 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:09,235 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:09,235 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,235 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:09,235 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:09,239 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,239 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:09,240 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:09,240 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:09,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:09,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ad8ba82[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:09,241 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:09,241 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:09,242 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:09,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:09,243 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:09,243 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:09,243 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:09,243 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,243 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:09,244 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:09,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:09,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:09,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:09,246 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:09,246 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:09,246 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:09,247 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:09,247 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:09,250 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:09,250 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:09,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:09,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:09,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:09,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:09,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:09,250 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:09,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:09,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:09,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:09,250 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:09,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:09,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:09,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:09,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,252 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:09,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:09,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,254 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:09,254 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:09,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:09,254 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:09,254 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:09,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:09,254 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:09,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:09,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:09,254 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:09,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:09,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:09,254 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:09,254 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:09,254 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,255 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:09,255 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:09,255 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:09,355 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:09,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:09,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:09,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:09,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:09,361 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:09,361 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:09,361 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:09,361 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:09,362 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:09,362 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:09,363 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:09,363 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:09,363 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:09,363 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:09,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:09,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:09,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:09,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:09,364 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:09,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:09,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:09,365 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:09,365 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:09,365 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:09,365 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:09,365 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:09,365 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:09,365 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:09,365 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-30 16:55:09,365 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:09,365 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:09,366 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:09,366 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:09,366 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:09,366 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:09,785 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033015f65013b2af4b8298e8075b64c0280f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033015f65013b2af4b8298e8075b64c0280f 2023-03-30 16:55:09,790 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=103, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/bd3ae11b9e4345e39eb7d7423592736b 2023-03-30 16:55:10,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:10,186 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:10,198 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/8592ff3476d54b628b12c8416a1b30b4 2023-03-30 16:55:10,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195370200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195370201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195370206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195370207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195370209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195370309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195370309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195370311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195370314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195370316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195370515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195370516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195370516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195370518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195370522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,610 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/bb7bec612ff94cc996aa5d9777ceadc5 2023-03-30 16:55:10,614 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/bd3ae11b9e4345e39eb7d7423592736b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/bd3ae11b9e4345e39eb7d7423592736b 2023-03-30 16:55:10,618 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/bd3ae11b9e4345e39eb7d7423592736b, entries=150, sequenceid=103, filesize=25.1 K 2023-03-30 16:55:10,619 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/8592ff3476d54b628b12c8416a1b30b4 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/8592ff3476d54b628b12c8416a1b30b4 2023-03-30 16:55:10,622 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/8592ff3476d54b628b12c8416a1b30b4, entries=150, sequenceid=103, filesize=11.6 K 2023-03-30 16:55:10,624 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/bb7bec612ff94cc996aa5d9777ceadc5 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/bb7bec612ff94cc996aa5d9777ceadc5 2023-03-30 16:55:10,630 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/bb7bec612ff94cc996aa5d9777ceadc5, entries=150, sequenceid=103, filesize=11.6 K 2023-03-30 16:55:10,631 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for e3fe0ff55782e6f5d2f55c861f91d3cb in 1266ms, sequenceid=103, compaction requested=true 2023-03-30 16:55:10,631 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:10,631 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:10,631 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:10,631 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:10,631 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,631 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:10,631 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:10,633 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:10,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:10,633 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:10,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:10,633 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:10,633 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:10,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:10,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:10,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:10,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:10,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7824295f[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:10,634 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:10,634 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:10,635 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:10,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:10,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:10,636 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,636 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:10,636 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:10,636 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:10,636 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:10,637 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:10,637 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:10,637 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,637 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:10,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:10,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:10,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:10,639 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:10,639 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:10,639 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:10,639 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:10,640 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:10,649 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:10,649 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:10,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:10,649 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:10,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:10,650 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:10,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:10,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:10,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:10,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:10,651 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,653 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,653 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:10,653 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:10,653 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:10,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:10,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:10,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:10,653 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:10,654 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:10,654 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:10,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:10,654 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:10,654 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:10,654 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:10,654 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:10,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:10,654 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:10,754 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:10,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:10,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:10,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:10,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:10,758 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:10,758 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:10,758 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:10,759 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:10,759 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:10,759 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:10,760 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,760 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,761 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:10,761 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:10,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:10,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:10,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:10,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:10,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:10,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:10,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:10,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:10,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:10,762 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:10,762 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:10,762 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:10,762 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:10,762 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:10,762 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:10,762 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:10,763 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-30 16:55:10,763 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:10,763 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:10,763 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:10,763 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:10,763 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:10,763 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:10,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:10,826 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:10,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195370842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195370844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195370844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195370845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195370845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195370946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195370953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195370953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195370953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:10,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:10,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195370954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:11,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195371153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:11,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195371161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:11,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195371161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:11,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195371161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:11,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195371162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,180 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303304dc7c8679fe740799376f8deadfcbe4c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303304dc7c8679fe740799376f8deadfcbe4c 2023-03-30 16:55:11,190 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=117, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/2dd0e65e8a08423387a1b84c49cbf021 2023-03-30 16:55:11,221 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/0e29618878cf466ca0656809be138503 2023-03-30 16:55:11,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:11,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195371460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:11,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195371466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:11,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195371466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:11,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195371467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:11,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195371468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,634 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/f1062cb1b71645a18fdc59fd517b32df 2023-03-30 16:55:11,638 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/2dd0e65e8a08423387a1b84c49cbf021 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/2dd0e65e8a08423387a1b84c49cbf021 2023-03-30 16:55:11,641 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/2dd0e65e8a08423387a1b84c49cbf021, entries=150, sequenceid=117, filesize=25.1 K 2023-03-30 16:55:11,642 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/0e29618878cf466ca0656809be138503 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/0e29618878cf466ca0656809be138503 2023-03-30 16:55:11,646 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/0e29618878cf466ca0656809be138503, entries=150, sequenceid=117, filesize=11.6 K 2023-03-30 16:55:11,648 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/f1062cb1b71645a18fdc59fd517b32df as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/f1062cb1b71645a18fdc59fd517b32df 2023-03-30 16:55:11,653 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/f1062cb1b71645a18fdc59fd517b32df, entries=150, sequenceid=117, filesize=11.6 K 2023-03-30 16:55:11,654 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for e3fe0ff55782e6f5d2f55c861f91d3cb in 892ms, sequenceid=117, compaction requested=true 2023-03-30 16:55:11,654 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:11,654 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:11,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:11,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:11,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:11,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:11,656 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,656 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:11,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:11,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:11,656 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:11,656 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:11,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:11,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:11,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:11,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:11,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:11,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@88bc897[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:11,657 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:11,658 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:11,659 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:11,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:11,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:11,659 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:11,659 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:11,660 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:11,660 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,660 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:11,661 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:11,661 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:11,661 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,661 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:11,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:11,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:11,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:11,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:11,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:11,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:11,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:11,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:11,663 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:11,663 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:11,663 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:11,663 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:11,663 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:11,665 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:11,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:11,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:11,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:11,665 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:11,665 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:11,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:11,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:11,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:11,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:11,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:11,665 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:11,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:11,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:11,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:11,666 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,669 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:11,669 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,669 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:11,669 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:11,669 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:11,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:11,669 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:11,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:11,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:11,669 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,669 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:11,669 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:11,669 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:11,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:11,669 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:11,669 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:11,770 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:11,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:11,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:11,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:11,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:11,773 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:11,773 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:11,773 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:11,773 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:11,774 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:11,774 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:11,776 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,776 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:11,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:11,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:11,776 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:11,776 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:11,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:11,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:11,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:11,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:11,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:11,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:11,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:11,777 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:11,777 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:11,777 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:11,777 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:11,777 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:11,777 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:11,777 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:11,778 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-30 16:55:11,778 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:11,778 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:11,778 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:11,778 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:11,778 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:11,778 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:11,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:11,981 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:11,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:11,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195371991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:12,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:12,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195371996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:12,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:12,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195371996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:12,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:12,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195371997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:12,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:12,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195371998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:12,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:12,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195372099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:12,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:12,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195372107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:12,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:12,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195372108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:12,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:12,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195372108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:12,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:12,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195372108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:12,191 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330249a4a7f90114e12b88f287cf7debef6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330249a4a7f90114e12b88f287cf7debef6 2023-03-30 16:55:12,201 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=135, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/6b380132a6874711a5cede0698af0058 2023-03-30 16:55:12,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:12,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195372308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:12,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:12,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195372315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:12,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:12,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195372315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:12,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:12,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195372315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:12,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:12,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195372315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:12,614 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/baa484a67ea047caa0358af1ea77d21c 2023-03-30 16:55:12,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:12,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195372616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:12,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:12,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195372626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:12,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:12,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195372627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:12,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:12,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195372627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:12,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:12,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195372627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,025 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/63d1445b855a43ea970672a6bac76243 2023-03-30 16:55:13,030 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/6b380132a6874711a5cede0698af0058 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/6b380132a6874711a5cede0698af0058 2023-03-30 16:55:13,034 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/6b380132a6874711a5cede0698af0058, entries=150, sequenceid=135, filesize=25.3 K 2023-03-30 16:55:13,035 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/baa484a67ea047caa0358af1ea77d21c as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/baa484a67ea047caa0358af1ea77d21c 2023-03-30 16:55:13,041 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/baa484a67ea047caa0358af1ea77d21c, entries=150, sequenceid=135, filesize=11.8 K 2023-03-30 16:55:13,044 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/63d1445b855a43ea970672a6bac76243 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/63d1445b855a43ea970672a6bac76243 2023-03-30 16:55:13,050 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/63d1445b855a43ea970672a6bac76243, entries=150, sequenceid=135, filesize=11.8 K 2023-03-30 16:55:13,051 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for e3fe0ff55782e6f5d2f55c861f91d3cb in 1273ms, sequenceid=135, compaction requested=true 2023-03-30 16:55:13,051 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:13,051 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:13,051 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:13,051 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:13,052 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,052 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:13,052 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:13,058 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,058 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,059 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,059 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:13,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e2bc006[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:13,060 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:13,061 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,062 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,062 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:13,062 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:13,062 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,062 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:13,062 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:13,068 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:13,068 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:13,068 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,068 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:13,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:13,072 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:13,072 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:13,072 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:13,072 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:13,073 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:13,074 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,074 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:13,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,075 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,075 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,076 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,079 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:13,079 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:13,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:13,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:13,079 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:13,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:13,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:13,079 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:13,079 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:13,079 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,079 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:13,079 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,079 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,079 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,079 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,079 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:13,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:13,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:13,129 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-30 16:55:13,129 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:13,129 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:13,129 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:13,129 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:13,130 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:13,130 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:13,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:13,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195373162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:13,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195373162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:13,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195373163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,179 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:13,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:13,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:13,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:13,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:13,184 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:13,184 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:13,184 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:13,184 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:13,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:13,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195373174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:13,185 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195373174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,185 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,187 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:13,187 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,188 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,188 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:13,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:13,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:13,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:13,192 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:13,192 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:13,192 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:13,192 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:13,192 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:13,192 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:13,192 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:13,192 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:13,193 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:13,193 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:13,193 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:13,193 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,193 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:13,193 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:13,194 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,194 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,194 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,194 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:13,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@416867eb[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:13,196 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:13,196 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,197 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,197 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:13,198 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:13,198 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:13,198 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,198 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:13,199 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,199 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:13,199 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:13,199 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:13,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:13,205 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:13,205 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:13,205 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:13,205 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:13,205 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:13,208 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,208 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:13,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,208 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:13,209 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,214 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,216 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,217 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:13,217 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:13,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:13,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:13,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:13,217 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,217 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,217 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,217 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:13,217 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:13,217 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:13,217 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:13,217 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,217 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:13,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:13,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195373275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:13,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195373275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:13,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195373275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:13,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195373286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:13,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195373286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,317 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:13,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:13,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:13,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:13,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:13,321 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:13,321 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:13,321 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:13,321 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:13,321 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,321 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,322 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:13,322 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:13,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:13,323 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,323 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:13,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:13,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:13,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:13,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:13,324 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:13,324 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:13,324 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:13,324 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:13,324 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:13,324 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:13,324 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:13,324 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:13,324 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:13,324 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:13,324 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:13,324 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,324 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:13,324 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:13,325 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,325 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,326 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,326 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:13,327 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:13,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@185fd048[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:13,327 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,329 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,329 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:13,329 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:13,329 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:13,329 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,329 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:13,331 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:13,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,331 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:13,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:13,333 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:13,333 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:13,333 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:13,333 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:13,333 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:13,334 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,334 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,334 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:13,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:13,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:13,335 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,337 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,339 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,339 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,339 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,339 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:13,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:13,339 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:13,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:13,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:13,339 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,339 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:13,339 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,339 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:13,339 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:13,339 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:13,339 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:13,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:13,439 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:13,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:13,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:13,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:13,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:13,443 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:13,443 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:13,443 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:13,443 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:13,443 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,443 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,446 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,446 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:13,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:13,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:13,446 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,446 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:13,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:13,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:13,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:13,447 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:13,448 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:13,448 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:13,448 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:13,448 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:13,448 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:13,448 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:13,448 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:13,448 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:13,448 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:13,448 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:13,448 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,448 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:13,448 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:13,450 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,450 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,451 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,451 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:13,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ec66ca4[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:13,452 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:13,452 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,453 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,453 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:13,453 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:13,453 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:13,453 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,453 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:13,455 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:13,455 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:13,455 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,455 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:13,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:13,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:13,457 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:13,457 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:13,457 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:13,457 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:13,457 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:13,462 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,462 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,462 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:13,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,463 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:13,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:13,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,464 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,472 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:13,472 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:13,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:13,472 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:13,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:13,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:13,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:13,472 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:13,472 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,473 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,473 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,473 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:13,473 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:13,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:13,473 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,473 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:13,473 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:13,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195373489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:13,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195373489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:13,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195373489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:13,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195373493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:13,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195373493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,569 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330c003e20b9e1042989d60cc3f80773abc to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330c003e20b9e1042989d60cc3f80773abc 2023-03-30 16:55:13,573 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:13,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:13,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:13,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:13,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:13,578 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:13,579 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:13,579 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:13,579 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:13,579 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,579 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,580 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,580 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:13,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:13,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:13,580 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,580 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:13,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:13,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:13,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:13,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:13,581 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:13,581 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:13,581 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:13,582 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:13,582 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:13,582 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:13,582 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:13,582 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:13,582 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:13,582 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:13,582 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:13,582 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,582 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:13,582 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:13,583 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,583 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,583 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,584 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:13,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ebdbe8a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:13,585 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:13,585 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,587 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,587 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:13,587 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:13,587 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:13,588 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,588 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:13,589 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:13,589 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,589 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:13,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,589 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:13,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:13,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:13,591 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:13,592 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:13,592 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:13,592 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:13,592 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:13,593 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,593 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,593 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:13,593 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,594 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:13,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:13,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,596 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,596 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,596 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:13,596 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:13,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,596 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:13,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:13,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:13,596 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,597 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,597 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,597 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,597 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:13,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:13,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:13,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:13,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:13,597 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:13,597 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:13,697 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:13,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:13,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:13,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:13,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:13,701 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:13,701 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:13,701 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:13,701 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:13,706 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,706 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,708 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,708 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:13,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:13,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:13,708 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,708 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:13,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:13,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:13,710 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:13,710 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:13,710 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:13,716 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:13,716 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:13,716 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:13,716 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:13,716 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:13,716 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:13,716 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:13,716 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:13,716 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,716 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:13,716 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:13,718 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,718 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,718 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,719 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:13,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:13,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ce8d589[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:13,720 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:13,720 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,721 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,722 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:13,722 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:13,722 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:13,722 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,722 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:13,723 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,723 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:13,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,723 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:13,723 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:13,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:13,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:13,726 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:13,726 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:13,727 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:13,727 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:13,727 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:13,729 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,729 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,729 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:13,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:13,730 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,731 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:13,733 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:13,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:13,734 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,734 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:13,734 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,734 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:13,734 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:13,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:13,734 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,734 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:13,734 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,734 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:13,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:13,734 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:13,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:13,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195373796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:13,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195373798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:13,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195373798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:13,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195373803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:13,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195373803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,834 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:13,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:13,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:13,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:13,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:13,840 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:13,840 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:13,840 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:13,840 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:13,841 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,841 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,844 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:13,844 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:13,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:13,845 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,845 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:13,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:13,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:13,846 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:13,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:13,846 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:13,846 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:13,846 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:13,846 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:13,846 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:13,846 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:13,846 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:13,846 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:13,846 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:13,846 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:13,846 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:13,846 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,846 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:13,847 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:13,849 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,849 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,849 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,850 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:13,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ad39bcc[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:13,851 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:13,851 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,856 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,856 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:13,856 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:13,856 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:13,857 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,857 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:13,859 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,859 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,859 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:13,859 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:13,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:13,862 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:13,862 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:13,862 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:13,862 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:13,862 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:13,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:13,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:13,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:13,864 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:13,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:13,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:13,865 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:13,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:13,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,870 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:13,870 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:13,870 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:13,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:13,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:13,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:13,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:13,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:13,870 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:13,870 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:13,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:13,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:13,871 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,871 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,871 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,871 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,871 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,871 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:13,871 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,970 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:13,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:13,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:13,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:13,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:13,992 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:13,992 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:13,992 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=150, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/b2500d6cd3634edaa8a8165057876348 2023-03-30 16:55:13,992 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:13,992 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:13,993 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,993 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,994 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,994 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:13,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:13,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:13,995 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:13,995 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:13,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:13,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:13,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:13,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:14,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:14,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:14,004 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:14,004 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:14,004 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:14,008 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:14,008 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:14,008 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:14,008 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:14,008 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:14,008 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:14,008 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:14,009 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:14,009 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,009 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:14,009 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:14,010 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,010 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,011 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,011 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:14,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:14,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70d6b7e[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:14,013 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:14,013 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,014 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,014 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:14,014 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:14,014 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:14,014 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,014 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:14,017 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,017 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,017 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,017 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:14,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:14,019 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:14,019 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:14,019 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:14,019 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:14,019 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:14,020 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,020 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:14,020 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:14,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:14,021 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,022 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,024 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:14,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:14,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:14,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:14,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:14,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:14,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:14,025 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:14,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:14,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:14,025 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:14,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:14,125 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:14,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:14,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:14,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:14,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:14,129 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:14,129 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:14,129 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:14,129 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:14,129 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,129 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,131 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:14,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:14,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:14,131 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,131 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,131 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:14,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:14,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:14,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:14,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:14,136 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:14,136 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:14,136 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:14,136 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:14,136 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:14,136 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:14,137 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:14,137 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:14,137 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:14,137 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:14,137 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:14,137 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,137 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:14,137 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:14,140 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:14,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74b7dcfb[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:14,144 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:14,144 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,145 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:14,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:14,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:14,147 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,147 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:14,148 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,148 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:14,148 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,148 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:14,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:14,150 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:14,151 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:14,151 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:14,151 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:14,151 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:14,152 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,153 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,154 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,154 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,154 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:14,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:14,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:14,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,158 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:14,158 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:14,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:14,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:14,158 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:14,158 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:14,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:14,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:14,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:14,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:14,158 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,158 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,158 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,159 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,159 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,159 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,159 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:14,159 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:14,259 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:14,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:14,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:14,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:14,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:14,263 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:14,263 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:14,263 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:14,263 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:14,266 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,266 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,267 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:14,268 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,268 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,268 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:14,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:14,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:14,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:14,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:14,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:14,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:14,271 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:14,271 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:14,271 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:14,271 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:14,271 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:14,271 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:14,272 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:14,272 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:14,272 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:14,272 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:14,272 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:14,272 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,272 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:14,272 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:14,274 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,274 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,274 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,274 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:14,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ea90b79[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:14,277 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:14,277 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,278 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,279 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,279 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:14,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:14,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:14,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:14,282 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,282 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:14,282 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:14,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,282 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:14,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:14,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:14,286 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:14,286 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:14,286 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:14,286 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:14,287 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:14,288 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,288 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,288 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:14,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:14,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:14,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,289 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,291 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,293 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,293 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,293 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:14,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:14,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:14,293 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:14,294 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:14,293 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,293 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:14,294 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:14,294 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,294 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,294 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:14,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:14,294 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:14,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:14,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:14,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:14,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195374301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,320 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:14,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195374310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:14,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195374312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:14,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195374314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:14,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195374316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,394 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:14,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:14,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:14,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:14,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:14,397 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:14,397 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:14,397 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:14,397 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:14,398 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,398 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,403 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:14,403 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:14,403 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,403 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:14,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:14,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:14,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:14,405 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:14,405 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:14,405 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:14,405 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:14,405 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:14,405 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:14,406 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:14,406 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:14,406 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:14,406 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:14,406 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:14,406 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,406 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:14,406 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:14,408 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,408 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,408 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:14,409 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:14,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48c2f89b[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:14,412 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:14,412 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,413 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,413 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:14,413 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:14,413 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:14,413 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,413 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:14,414 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/47ed8b0684c64e62b27c0fcf186a3aea 2023-03-30 16:55:14,415 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,415 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:14,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,415 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:14,415 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:14,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:14,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:14,417 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:14,417 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:14,417 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:14,417 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:14,417 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:14,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:14,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:14,421 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,422 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,424 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,424 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,424 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,424 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:14,424 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:14,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:14,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:14,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:14,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:14,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:14,424 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,425 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,425 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,425 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,424 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:14,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:14,425 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:14,425 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:14,525 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:14,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:14,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:14,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:14,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:14,528 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:14,528 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:14,528 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:14,528 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:14,528 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,528 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,530 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:14,530 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:14,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:14,530 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,530 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:14,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:14,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:14,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:14,531 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:14,531 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:14,531 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:14,531 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:14,531 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:14,531 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:14,531 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:14,531 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:14,532 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:14,532 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:14,532 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:14,532 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,532 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:14,532 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:14,533 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,533 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,533 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,533 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:14,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:14,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24783c66[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:14,535 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:14,535 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,537 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,537 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:14,537 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:14,537 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:14,537 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,538 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:14,538 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,538 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:14,539 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,539 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:14,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:14,541 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:14,541 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:14,541 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:14,541 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:14,541 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:14,543 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,543 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,543 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:14,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,543 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,544 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,546 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,546 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,546 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:14,546 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:14,546 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,546 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:14,546 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:14,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:14,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:14,546 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:14,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:14,547 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,547 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,547 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:14,547 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:14,647 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:14,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:14,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:14,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:14,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:14,650 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:14,650 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:14,650 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:14,650 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:14,651 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,651 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,653 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:14,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:14,653 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,653 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,654 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:14,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:14,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:14,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:14,655 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:14,655 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:14,655 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:14,655 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:14,655 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:14,655 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:14,655 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:14,655 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:14,655 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,655 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:14,655 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:14,656 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,656 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,657 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,657 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:14,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ee27c16[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:14,658 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:14,658 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,660 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,660 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:14,666 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,666 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:14,666 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:14,666 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:14,667 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,667 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:14,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,667 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:14,667 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:14,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:14,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:14,669 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:14,669 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:14,669 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:14,669 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:14,669 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:14,671 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,671 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,671 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:14,671 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,672 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,674 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:14,674 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:14,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:14,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:14,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:14,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:14,674 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,674 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:14,674 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:14,674 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,674 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:14,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:14,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:14,674 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,675 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:14,675 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,675 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,675 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,775 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:14,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:14,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:14,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:14,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:14,778 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:14,778 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:14,778 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:14,778 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:14,778 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,778 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,780 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,780 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:14,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:14,781 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,781 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:14,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:14,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:14,782 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:14,782 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:14,782 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:14,782 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:14,782 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:14,786 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,786 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,787 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,787 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:14,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8764070[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:14,788 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:14,788 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,789 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,789 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:14,789 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:14,789 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:14,789 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,790 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:14,791 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,791 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,791 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:14,792 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:14,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:14,794 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:14,794 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:14,794 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:14,794 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:14,794 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:14,796 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,796 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:14,796 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:14,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:14,796 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:14,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,797 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:14,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:14,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,799 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,799 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,799 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:14,799 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:14,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:14,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:14,799 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:14,799 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:14,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:14,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:14,799 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:14,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:14,800 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,800 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:14,800 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:14,800 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,800 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:14,827 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/3266846abdc74af7b3716d7249f7075e 2023-03-30 16:55:14,831 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/b2500d6cd3634edaa8a8165057876348 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/b2500d6cd3634edaa8a8165057876348 2023-03-30 16:55:14,835 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/b2500d6cd3634edaa8a8165057876348, entries=200, sequenceid=150, filesize=32.1 K 2023-03-30 16:55:14,836 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/47ed8b0684c64e62b27c0fcf186a3aea as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/47ed8b0684c64e62b27c0fcf186a3aea 2023-03-30 16:55:14,840 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/47ed8b0684c64e62b27c0fcf186a3aea, entries=150, sequenceid=150, filesize=11.8 K 2023-03-30 16:55:14,841 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/3266846abdc74af7b3716d7249f7075e as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/3266846abdc74af7b3716d7249f7075e 2023-03-30 16:55:14,849 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/3266846abdc74af7b3716d7249f7075e, entries=150, sequenceid=150, filesize=11.8 K 2023-03-30 16:55:14,850 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for e3fe0ff55782e6f5d2f55c861f91d3cb in 1721ms, sequenceid=150, compaction requested=true 2023-03-30 16:55:14,850 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:14,850 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-30 16:55:14,851 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:14,851 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:14,851 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:14,851 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-30 16:55:14,853 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 187615 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-30 16:55:14,853 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): e3fe0ff55782e6f5d2f55c861f91d3cb/A is initiating minor compaction (all files) 2023-03-30 16:55:14,853 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e3fe0ff55782e6f5d2f55c861f91d3cb/A in TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:14,853 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/f3e258bc59f64f31b9c707026b0392ff, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/73564254ccd94243b88f67dcd35228fb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/6cf862c234024c48b523fe502089e003, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/bd3ae11b9e4345e39eb7d7423592736b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/2dd0e65e8a08423387a1b84c49cbf021, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/6b380132a6874711a5cede0698af0058, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/b2500d6cd3634edaa8a8165057876348] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp, totalSize=183.2 K 2023-03-30 16:55:14,853 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting f3e258bc59f64f31b9c707026b0392ff, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680195306416 2023-03-30 16:55:14,854 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83605 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-30 16:55:14,854 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): e3fe0ff55782e6f5d2f55c861f91d3cb/B is initiating minor compaction (all files) 2023-03-30 16:55:14,854 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 73564254ccd94243b88f67dcd35228fb, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1680195306762 2023-03-30 16:55:14,854 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3fe0ff55782e6f5d2f55c861f91d3cb/B in TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:14,854 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/e8bbd703c8fc408ab94f92d82cfdb81f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/3b301ea7019a4d499efda9139729978f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/c5d4d7e365b34a36963e766783856006, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/8592ff3476d54b628b12c8416a1b30b4, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/0e29618878cf466ca0656809be138503, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/baa484a67ea047caa0358af1ea77d21c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/47ed8b0684c64e62b27c0fcf186a3aea] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp, totalSize=81.6 K 2023-03-30 16:55:14,856 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting e8bbd703c8fc408ab94f92d82cfdb81f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680195306416 2023-03-30 16:55:14,856 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 6cf862c234024c48b523fe502089e003, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680195307401 2023-03-30 16:55:14,856 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 3b301ea7019a4d499efda9139729978f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1680195306762 2023-03-30 16:55:14,856 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting bd3ae11b9e4345e39eb7d7423592736b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680195308045 2023-03-30 16:55:14,856 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting c5d4d7e365b34a36963e766783856006, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680195307401 2023-03-30 16:55:14,856 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 2dd0e65e8a08423387a1b84c49cbf021, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680195310205 2023-03-30 16:55:14,857 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 8592ff3476d54b628b12c8416a1b30b4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680195308045 2023-03-30 16:55:14,857 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 6b380132a6874711a5cede0698af0058, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1680195310843 2023-03-30 16:55:14,857 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting b2500d6cd3634edaa8a8165057876348, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680195311993 2023-03-30 16:55:14,857 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 0e29618878cf466ca0656809be138503, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680195310205 2023-03-30 16:55:14,858 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting baa484a67ea047caa0358af1ea77d21c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1680195310843 2023-03-30 16:55:14,858 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 47ed8b0684c64e62b27c0fcf186a3aea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680195311993 2023-03-30 16:55:14,900 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:14,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:14,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:14,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:14,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:14,908 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:14,908 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:14,908 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:14,908 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:14,909 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,909 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,910 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,911 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:14,911 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:14,911 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:14,911 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): e3fe0ff55782e6f5d2f55c861f91d3cb#B#compaction#303 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:14,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:14,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:14,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:14,912 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:14,918 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): e3fe0ff55782e6f5d2f55c861f91d3cb#A#compaction#302 average throughput is 0.53 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:14,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:14,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:14,928 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:14,928 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:14,928 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:14,936 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:14,936 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:14,936 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:14,936 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:14,936 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-30 16:55:14,937 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:14,937 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:14,937 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:14,937 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:14,937 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:14,937 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:14,985 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303308bfd9c67802041388ed9df6d780bc9df to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303308bfd9c67802041388ed9df6d780bc9df 2023-03-30 16:55:15,007 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=168, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/c13098f4a81e475f9a8fc52e24055c0f 2023-03-30 16:55:15,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:15,323 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:15,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195375338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195375342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195375343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195375344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195375345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,369 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/4d0838dce9174507972ca85a717093bb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/4d0838dce9174507972ca85a717093bb 2023-03-30 16:55:15,373 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in e3fe0ff55782e6f5d2f55c861f91d3cb/B of e3fe0ff55782e6f5d2f55c861f91d3cb into 4d0838dce9174507972ca85a717093bb(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:15,373 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:15,373 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb., storeName=e3fe0ff55782e6f5d2f55c861f91d3cb/B, priority=9, startTime=1680195314851; duration=0sec 2023-03-30 16:55:15,373 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:15,373 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-30 16:55:15,375 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83605 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-30 16:55:15,375 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): e3fe0ff55782e6f5d2f55c861f91d3cb/C is initiating minor compaction (all files) 2023-03-30 16:55:15,376 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3fe0ff55782e6f5d2f55c861f91d3cb/C in TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:15,376 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/6bb362391f05459babc371369e211283, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/25aa9583403e4a35a21fddac5cdd89b3, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/a22a061eee32485d9cd4341fbb8444d5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/bb7bec612ff94cc996aa5d9777ceadc5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/f1062cb1b71645a18fdc59fd517b32df, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/63d1445b855a43ea970672a6bac76243, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/3266846abdc74af7b3716d7249f7075e] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp, totalSize=81.6 K 2023-03-30 16:55:15,377 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 6bb362391f05459babc371369e211283, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680195306416 2023-03-30 16:55:15,378 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 25aa9583403e4a35a21fddac5cdd89b3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1680195306762 2023-03-30 16:55:15,378 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting a22a061eee32485d9cd4341fbb8444d5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680195307401 2023-03-30 16:55:15,378 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting bb7bec612ff94cc996aa5d9777ceadc5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680195308045 2023-03-30 16:55:15,379 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting f1062cb1b71645a18fdc59fd517b32df, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680195310205 2023-03-30 16:55:15,379 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 63d1445b855a43ea970672a6bac76243, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1680195310843 2023-03-30 16:55:15,379 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 3266846abdc74af7b3716d7249f7075e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680195311993 2023-03-30 16:55:15,389 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/a24a78ec3b404186aee26b342d86f53e as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/a24a78ec3b404186aee26b342d86f53e 2023-03-30 16:55:15,390 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): e3fe0ff55782e6f5d2f55c861f91d3cb#C#compaction#306 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:15,394 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in e3fe0ff55782e6f5d2f55c861f91d3cb/A of e3fe0ff55782e6f5d2f55c861f91d3cb into a24a78ec3b404186aee26b342d86f53e(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:15,394 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:15,394 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb., storeName=e3fe0ff55782e6f5d2f55c861f91d3cb/A, priority=9, startTime=1680195314850; duration=0sec 2023-03-30 16:55:15,394 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:15,406 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/0dc79afe64b247d68bf8c28df73108f8 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/0dc79afe64b247d68bf8c28df73108f8 2023-03-30 16:55:15,423 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/1c1f1f9bcffb4721928f6d8e412e8dc8 2023-03-30 16:55:15,425 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in e3fe0ff55782e6f5d2f55c861f91d3cb/C of e3fe0ff55782e6f5d2f55c861f91d3cb into 0dc79afe64b247d68bf8c28df73108f8(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:15,425 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:15,425 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb., storeName=e3fe0ff55782e6f5d2f55c861f91d3cb/C, priority=9, startTime=1680195314851; duration=0sec 2023-03-30 16:55:15,425 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:15,447 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/4c70867ab20143bc9c008d7ebac151d9 2023-03-30 16:55:15,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195375446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195375447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,467 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/c13098f4a81e475f9a8fc52e24055c0f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c13098f4a81e475f9a8fc52e24055c0f 2023-03-30 16:55:15,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195375463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195375463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,471 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c13098f4a81e475f9a8fc52e24055c0f, entries=150, sequenceid=168, filesize=25.3 K 2023-03-30 16:55:15,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195375464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,474 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/1c1f1f9bcffb4721928f6d8e412e8dc8 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/1c1f1f9bcffb4721928f6d8e412e8dc8 2023-03-30 16:55:15,479 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/1c1f1f9bcffb4721928f6d8e412e8dc8, entries=150, sequenceid=168, filesize=11.8 K 2023-03-30 16:55:15,480 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/4c70867ab20143bc9c008d7ebac151d9 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/4c70867ab20143bc9c008d7ebac151d9 2023-03-30 16:55:15,483 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/4c70867ab20143bc9c008d7ebac151d9, entries=150, sequenceid=168, filesize=11.8 K 2023-03-30 16:55:15,484 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for e3fe0ff55782e6f5d2f55c861f91d3cb in 548ms, sequenceid=168, compaction requested=false 2023-03-30 16:55:15,484 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:15,484 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:15,484 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:15,484 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:15,484 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,484 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:15,484 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:15,487 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,487 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:15,488 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:15,488 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:15,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:15,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@572f67c5[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:15,489 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:15,489 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:15,491 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:15,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:15,492 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:15,492 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:15,492 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:15,492 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,492 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:15,493 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:15,493 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:15,493 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,493 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:15,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:15,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:15,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:15,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:15,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:15,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:15,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:15,499 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:15,499 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:15,499 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:15,499 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:15,499 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:15,501 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:15,501 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:15,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:15,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:15,501 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:15,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:15,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:15,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:15,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:15,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:15,501 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:15,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:15,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:15,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:15,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:15,502 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:15,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:15,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,506 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,506 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:15,506 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:15,506 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:15,506 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:15,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:15,506 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,506 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:15,506 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:15,506 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:15,506 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:15,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:15,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:15,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:15,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:15,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:15,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:15,506 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:15,507 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:15,607 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:15,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:15,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:15,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:15,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:15,610 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:15,610 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:15,610 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:15,611 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:15,611 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:15,611 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:15,612 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,613 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:15,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:15,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:15,613 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,613 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:15,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:15,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:15,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:15,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:15,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:15,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:15,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:15,614 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:15,614 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:15,614 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:15,614 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-30 16:55:15,615 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:15,615 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:15,615 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:15,615 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:15,615 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:15,615 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:15,637 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303301a9d73cc31c1445a86179baf316d5555 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303301a9d73cc31c1445a86179baf316d5555 2023-03-30 16:55:15,643 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=185, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/c72a3ae0dc9e4ce38f1f87e13fe182df 2023-03-30 16:55:15,654 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/a65302887b5d4feab61c2ac7e4bd2897 2023-03-30 16:55:15,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:15,654 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:15,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195375679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195375682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195375683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195375683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195375684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195375788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195375789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195375789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195375789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195375790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195375993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195375993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195375994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:15,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:15,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195375994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195376000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,064 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/8fcdbc587c68401a846dd7424bfff6b5 2023-03-30 16:55:16,068 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/c72a3ae0dc9e4ce38f1f87e13fe182df as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c72a3ae0dc9e4ce38f1f87e13fe182df 2023-03-30 16:55:16,071 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c72a3ae0dc9e4ce38f1f87e13fe182df, entries=150, sequenceid=185, filesize=25.3 K 2023-03-30 16:55:16,072 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/a65302887b5d4feab61c2ac7e4bd2897 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/a65302887b5d4feab61c2ac7e4bd2897 2023-03-30 16:55:16,075 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/a65302887b5d4feab61c2ac7e4bd2897, entries=150, sequenceid=185, filesize=11.8 K 2023-03-30 16:55:16,075 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/8fcdbc587c68401a846dd7424bfff6b5 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/8fcdbc587c68401a846dd7424bfff6b5 2023-03-30 16:55:16,078 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/8fcdbc587c68401a846dd7424bfff6b5, entries=150, sequenceid=185, filesize=11.8 K 2023-03-30 16:55:16,079 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for e3fe0ff55782e6f5d2f55c861f91d3cb in 465ms, sequenceid=185, compaction requested=true 2023-03-30 16:55:16,079 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:16,079 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:16,079 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:16,079 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:16,079 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,079 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:16,079 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:16,081 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:16,081 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:16,081 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:16,081 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:16,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:16,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32d1fa4a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:16,083 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:16,083 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:16,084 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:16,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:16,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:16,085 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:16,085 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:16,085 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,085 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:16,085 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:16,092 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:16,092 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,092 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:16,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:16,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:16,092 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:16,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:16,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:16,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:16,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:16,094 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:16,095 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:16,095 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:16,095 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:16,095 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:16,096 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:16,096 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:16,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:16,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:16,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:16,096 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:16,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:16,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:16,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:16,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:16,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:16,097 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:16,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:16,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:16,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:16,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,098 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:16,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:16,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,100 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,100 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:16,100 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:16,100 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:16,100 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:16,101 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,101 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:16,101 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:16,101 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:16,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:16,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:16,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:16,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:16,100 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:16,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:16,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:16,101 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:16,101 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:16,201 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:16,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:16,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:16,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:16,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:16,204 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:16,204 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:16,204 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:16,204 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:16,205 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:16,205 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:16,206 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,206 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:16,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:16,206 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,206 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:16,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:16,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:16,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:16,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:16,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:16,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:16,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:16,207 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:16,207 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:16,207 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:16,208 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:16,208 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:16,208 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:16,208 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:16,208 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-30 16:55:16,208 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:16,208 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:16,208 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:16,208 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:16,208 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:16,208 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:16,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:16,322 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:16,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195376331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195376332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195376334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195376338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195376339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195376439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195376439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195376440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195376440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195376446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,623 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330d7e79a11485b44678fb2ffe4c7328927 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d7e79a11485b44678fb2ffe4c7328927 2023-03-30 16:55:16,631 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=203, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/c994cbd7f0eb400ca2f9691e099759e0 2023-03-30 16:55:16,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195376644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195376644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195376644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195376645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,657 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/a83ac8e06e644b1f9cf0e88d47d3f3e8 2023-03-30 16:55:16,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195376652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,676 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/3909d2f732834a86b0057b898bc75b5b 2023-03-30 16:55:16,681 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/c994cbd7f0eb400ca2f9691e099759e0 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c994cbd7f0eb400ca2f9691e099759e0 2023-03-30 16:55:16,687 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c994cbd7f0eb400ca2f9691e099759e0, entries=150, sequenceid=203, filesize=25.3 K 2023-03-30 16:55:16,688 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/a83ac8e06e644b1f9cf0e88d47d3f3e8 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/a83ac8e06e644b1f9cf0e88d47d3f3e8 2023-03-30 16:55:16,695 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/a83ac8e06e644b1f9cf0e88d47d3f3e8, entries=150, sequenceid=203, filesize=11.8 K 2023-03-30 16:55:16,696 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/3909d2f732834a86b0057b898bc75b5b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/3909d2f732834a86b0057b898bc75b5b 2023-03-30 16:55:16,700 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/3909d2f732834a86b0057b898bc75b5b, entries=150, sequenceid=203, filesize=11.8 K 2023-03-30 16:55:16,701 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for e3fe0ff55782e6f5d2f55c861f91d3cb in 493ms, sequenceid=203, compaction requested=true 2023-03-30 16:55:16,701 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:16,701 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:16,701 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:16,701 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:16,701 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,702 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:16,702 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:16,706 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:16,706 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:16,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:16,707 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:16,707 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:16,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:16,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:16,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:16,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:16,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:16,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fae32ca[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:16,709 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:16,709 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:16,711 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:16,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:16,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:16,712 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:16,712 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:16,712 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:16,712 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,712 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:16,717 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:16,717 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:16,718 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:16,719 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:16,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:16,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:16,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:16,723 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:16,723 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:16,723 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:16,723 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:16,723 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:16,726 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:16,726 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:16,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:16,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:16,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:16,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:16,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:16,726 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:16,727 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:16,728 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,730 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:16,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:16,730 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:16,731 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:16,731 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:16,730 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:16,731 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:16,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:16,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:16,730 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:16,731 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:16,731 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,731 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:16,731 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:16,731 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:16,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:16,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:16,831 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:16,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:16,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:16,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:16,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:16,834 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:16,834 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:16,834 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:16,834 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:16,835 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:16,835 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:16,836 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:16,836 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:16,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:16,836 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,836 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:16,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:16,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:16,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:16,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:16,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:16,837 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:16,837 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:16,837 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:16,837 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:16,837 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:16,837 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:16,837 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:16,838 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-30 16:55:16,838 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:16,838 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:16,838 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:16,838 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:16,838 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:16,838 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:16,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:16,953 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:16,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195376972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195376973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195376976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195376977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:16,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:16,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195376978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:17,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:17,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195377078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:17,091 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:17,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195377084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:17,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:17,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195377085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:17,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:17,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60884 deadline: 1680195377086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:17,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:17,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195377086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:17,254 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303301a7ca0bf97514beaaa972bff2a1a0f8f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303301a7ca0bf97514beaaa972bff2a1a0f8f 2023-03-30 16:55:17,289 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:17,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195377286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:17,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:17,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195377293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:17,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:17,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195377294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:17,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:17,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195377294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:17,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:17,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60926 deadline: 1680195377294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:17,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:17,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195377591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:17,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:17,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195377599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:17,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:17,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195377600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:17,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:17,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60866 deadline: 1680195377600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:17,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:17,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195377601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:17,659 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=218, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/3d4e10d342b2442d9f59b81d2b91d423 2023-03-30 16:55:18,080 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/1154dccb2bb84992a3421a7ee89945a7 2023-03-30 16:55:18,090 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/cde10d4c9ee04482a7df5929b68fb75a 2023-03-30 16:55:18,098 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/3d4e10d342b2442d9f59b81d2b91d423 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/3d4e10d342b2442d9f59b81d2b91d423 2023-03-30 16:55:18,101 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/3d4e10d342b2442d9f59b81d2b91d423, entries=150, sequenceid=218, filesize=25.3 K 2023-03-30 16:55:18,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60892 deadline: 1680195378098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,102 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/1154dccb2bb84992a3421a7ee89945a7 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/1154dccb2bb84992a3421a7ee89945a7 2023-03-30 16:55:18,107 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/1154dccb2bb84992a3421a7ee89945a7, entries=150, sequenceid=218, filesize=11.8 K 2023-03-30 16:55:18,107 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/cde10d4c9ee04482a7df5929b68fb75a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/cde10d4c9ee04482a7df5929b68fb75a 2023-03-30 16:55:18,112 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/cde10d4c9ee04482a7df5929b68fb75a, entries=150, sequenceid=218, filesize=11.8 K 2023-03-30 16:55:18,113 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for e3fe0ff55782e6f5d2f55c861f91d3cb in 1276ms, sequenceid=218, compaction requested=true 2023-03-30 16:55:18,113 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:18,113 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:18,113 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:18,113 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:18,113 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,113 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:18,113 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:18,115 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,115 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:18,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:18,115 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,115 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:18,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:18,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:18,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:18,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:18,117 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:18,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75618e8f[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:18,118 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:18,119 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,119 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-30 16:55:18,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,120 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:18,120 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:18,120 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:18,120 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,120 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:18,121 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:18,121 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:18,121 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:18,121 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:18,121 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:18,121 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:18,121 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:18,122 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:18,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:18,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:18,128 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:18,128 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:18,128 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:18,128 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:18,128 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:18,130 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,130 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:18,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:18,130 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:18,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:18,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:18,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:18,131 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,133 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,135 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,135 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:18,135 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:18,135 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:18,135 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:18,136 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,136 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,136 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,136 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:18,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:18,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:18,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:18,135 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:18,136 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:18,136 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:18,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:18,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:18,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:18,140 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330708d75b02adf4bd589a394ff5433ae7b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330708d75b02adf4bd589a394ff5433ae7b 2023-03-30 16:55:18,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60908 deadline: 1680195378136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,147 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=237, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/619c0dbe2aae48f591ab41725016137d 2023-03-30 16:55:18,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195378143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195378145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195378145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,236 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:18,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:18,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:18,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:18,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:18,239 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:18,239 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:18,239 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:18,239 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:18,240 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,240 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:18,242 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,242 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:18,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:18,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:18,243 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,243 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:18,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:18,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:18,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:18,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:18,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:18,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:18,244 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:18,244 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:18,244 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:18,244 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:18,246 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,246 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:18,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:18,246 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,246 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:18,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:18,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:18,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:18,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:18,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@595ccea9[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:18,248 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:18,248 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,249 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,249 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:18,249 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:18,249 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,249 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:18,249 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:18,250 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,250 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:18,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:18,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:18,250 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:18,250 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:18,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:18,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:18,252 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:18,253 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:18,253 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:18,253 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:18,253 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:18,255 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,255 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,255 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:18,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:18,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:18,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:18,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:18,256 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:18,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:18,257 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:18,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,258 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,258 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:18,258 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:18,258 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:18,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:18,258 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:18,258 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:18,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:18,259 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:18,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:18,259 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,259 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,259 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,259 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:18,259 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:18,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:18,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:18,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:18,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60908 deadline: 1680195378253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195378254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195378255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195378255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,359 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:18,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:18,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:18,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:18,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:18,362 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:18,362 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:18,362 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:18,362 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:18,363 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,363 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:18,365 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:18,365 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:18,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:18,365 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,365 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:18,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:18,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:18,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:18,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:18,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:18,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:18,366 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:18,366 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:18,366 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:18,368 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:18,368 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:18,368 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:18,368 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:18,368 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:18,368 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:18,368 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:18,368 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:18,368 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,368 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:18,368 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:18,369 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,369 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:18,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:18,370 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,370 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:18,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:18,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:18,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:18,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:18,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64e27c62[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:18,371 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:18,371 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,374 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:18,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:18,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:18,374 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,374 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:18,376 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:18,376 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:18,376 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:18,376 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:18,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:18,378 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:18,378 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:18,378 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:18,378 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:18,378 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:18,383 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,383 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,383 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:18,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:18,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:18,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:18,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:18,383 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:18,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:18,385 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:18,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,389 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,389 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:18,389 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:18,389 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,389 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,389 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,389 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,389 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:18,389 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:18,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:18,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:18,389 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:18,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:18,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:18,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:18,390 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:18,390 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:18,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:18,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:18,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60908 deadline: 1680195378462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195378463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195378463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195378463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,490 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:18,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:18,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:18,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:18,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:18,493 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:18,493 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:18,493 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:18,493 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:18,494 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,494 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:18,495 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:18,495 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:18,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:18,495 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,495 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:18,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:18,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:18,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:18,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:18,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:18,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:18,496 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:18,496 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:18,496 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:18,496 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:18,496 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:18,497 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:18,497 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:18,497 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:18,497 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:18,497 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:18,497 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:18,497 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,497 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:18,497 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:18,498 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,498 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:18,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:18,498 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,498 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:18,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:18,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ae42a56[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:18,500 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:18,500 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,501 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,501 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:18,501 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:18,501 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:18,503 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:18,503 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,503 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:18,504 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:18,504 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:18,504 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:18,504 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:18,504 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:18,504 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:18,504 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:18,504 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:18,505 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,505 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,505 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:18,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:18,506 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,507 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:18,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:18,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:18,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:18,509 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:18,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:18,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:18,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:18,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:18,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:18,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:18,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:18,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:18,509 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:18,510 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:18,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:18,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:18,563 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/2dd9d0025465449495f1616f4319ce82 2023-03-30 16:55:18,573 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/e92b71aa13604e8682ff93b5516345fb 2023-03-30 16:55:18,576 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/619c0dbe2aae48f591ab41725016137d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/619c0dbe2aae48f591ab41725016137d 2023-03-30 16:55:18,580 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/619c0dbe2aae48f591ab41725016137d, entries=250, sequenceid=237, filesize=38.9 K 2023-03-30 16:55:18,581 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/2dd9d0025465449495f1616f4319ce82 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/2dd9d0025465449495f1616f4319ce82 2023-03-30 16:55:18,585 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/2dd9d0025465449495f1616f4319ce82, entries=150, sequenceid=237, filesize=11.8 K 2023-03-30 16:55:18,589 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/e92b71aa13604e8682ff93b5516345fb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/e92b71aa13604e8682ff93b5516345fb 2023-03-30 16:55:18,592 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/e92b71aa13604e8682ff93b5516345fb, entries=150, sequenceid=237, filesize=11.8 K 2023-03-30 16:55:18,593 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for e3fe0ff55782e6f5d2f55c861f91d3cb in 474ms, sequenceid=237, compaction requested=true 2023-03-30 16:55:18,593 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:18,593 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:18,594 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-30 16:55:18,595 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-30 16:55:18,595 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:18,596 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:18,599 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 169702 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-30 16:55:18,599 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): e3fe0ff55782e6f5d2f55c861f91d3cb/A is initiating minor compaction (all files) 2023-03-30 16:55:18,599 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72558 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-30 16:55:18,599 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e3fe0ff55782e6f5d2f55c861f91d3cb/A in TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:18,599 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): e3fe0ff55782e6f5d2f55c861f91d3cb/B is initiating minor compaction (all files) 2023-03-30 16:55:18,599 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/a24a78ec3b404186aee26b342d86f53e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c13098f4a81e475f9a8fc52e24055c0f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c72a3ae0dc9e4ce38f1f87e13fe182df, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c994cbd7f0eb400ca2f9691e099759e0, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/3d4e10d342b2442d9f59b81d2b91d423, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/619c0dbe2aae48f591ab41725016137d] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp, totalSize=165.7 K 2023-03-30 16:55:18,599 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3fe0ff55782e6f5d2f55c861f91d3cb/B in TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:18,599 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/4d0838dce9174507972ca85a717093bb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/1c1f1f9bcffb4721928f6d8e412e8dc8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/a65302887b5d4feab61c2ac7e4bd2897, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/a83ac8e06e644b1f9cf0e88d47d3f3e8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/1154dccb2bb84992a3421a7ee89945a7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/2dd9d0025465449495f1616f4319ce82] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp, totalSize=70.9 K 2023-03-30 16:55:18,600 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting a24a78ec3b404186aee26b342d86f53e, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680195311993 2023-03-30 16:55:18,600 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 4d0838dce9174507972ca85a717093bb, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680195311993 2023-03-30 16:55:18,600 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 1c1f1f9bcffb4721928f6d8e412e8dc8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680195313159 2023-03-30 16:55:18,600 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting c13098f4a81e475f9a8fc52e24055c0f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680195313159 2023-03-30 16:55:18,600 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting a65302887b5d4feab61c2ac7e4bd2897, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680195315341 2023-03-30 16:55:18,601 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting c72a3ae0dc9e4ce38f1f87e13fe182df, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680195315341 2023-03-30 16:55:18,601 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting a83ac8e06e644b1f9cf0e88d47d3f3e8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680195315682 2023-03-30 16:55:18,601 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting c994cbd7f0eb400ca2f9691e099759e0, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680195315682 2023-03-30 16:55:18,601 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 1154dccb2bb84992a3421a7ee89945a7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680195316330 2023-03-30 16:55:18,601 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 3d4e10d342b2442d9f59b81d2b91d423, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680195316330 2023-03-30 16:55:18,601 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 2dd9d0025465449495f1616f4319ce82, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680195316972 2023-03-30 16:55:18,602 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 619c0dbe2aae48f591ab41725016137d, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680195316972 2023-03-30 16:55:18,610 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:18,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:18,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:18,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:18,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:18,616 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:18,616 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:18,616 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:18,616 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:18,616 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,616 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:18,618 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:18,618 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:18,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:18,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:18,619 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,619 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:18,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:18,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:18,623 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): e3fe0ff55782e6f5d2f55c861f91d3cb#B#compaction#320 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:18,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:18,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:18,627 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:18,627 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:18,627 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:18,627 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:18,627 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-30 16:55:18,627 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:18,628 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:18,628 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:18,628 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:18,628 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:18,628 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:18,631 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): e3fe0ff55782e6f5d2f55c861f91d3cb#A#compaction#321 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:18,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:18,772 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:18,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195378809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195378810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195378811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60908 deadline: 1680195378817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195378918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195378924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195378924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:18,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:18,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60908 deadline: 1680195378925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,036 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/381daa52fdc242b59ccda3fe537cf08f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/381daa52fdc242b59ccda3fe537cf08f 2023-03-30 16:55:19,041 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e3fe0ff55782e6f5d2f55c861f91d3cb/B of e3fe0ff55782e6f5d2f55c861f91d3cb into 381daa52fdc242b59ccda3fe537cf08f(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:19,041 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:19,041 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb., storeName=e3fe0ff55782e6f5d2f55c861f91d3cb/B, priority=10, startTime=1680195318595; duration=0sec 2023-03-30 16:55:19,041 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:19,041 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-30 16:55:19,046 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72558 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-30 16:55:19,046 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): e3fe0ff55782e6f5d2f55c861f91d3cb/C is initiating minor compaction (all files) 2023-03-30 16:55:19,046 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3fe0ff55782e6f5d2f55c861f91d3cb/C in TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:19,046 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/0dc79afe64b247d68bf8c28df73108f8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/4c70867ab20143bc9c008d7ebac151d9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/8fcdbc587c68401a846dd7424bfff6b5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/3909d2f732834a86b0057b898bc75b5b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/cde10d4c9ee04482a7df5929b68fb75a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/e92b71aa13604e8682ff93b5516345fb] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp, totalSize=70.9 K 2023-03-30 16:55:19,048 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 0dc79afe64b247d68bf8c28df73108f8, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680195311993 2023-03-30 16:55:19,048 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 4c70867ab20143bc9c008d7ebac151d9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680195313159 2023-03-30 16:55:19,049 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 8fcdbc587c68401a846dd7424bfff6b5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1680195315341 2023-03-30 16:55:19,049 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 3909d2f732834a86b0057b898bc75b5b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680195315682 2023-03-30 16:55:19,049 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting cde10d4c9ee04482a7df5929b68fb75a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680195316330 2023-03-30 16:55:19,050 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting e92b71aa13604e8682ff93b5516345fb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680195316972 2023-03-30 16:55:19,050 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330dc5d013dd806417c8eaea72472398017 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330dc5d013dd806417c8eaea72472398017 2023-03-30 16:55:19,057 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=250, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/828e1447ec224afe8f962ebea27626ef 2023-03-30 16:55:19,059 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): e3fe0ff55782e6f5d2f55c861f91d3cb#C#compaction#323 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:19,071 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/7eb2667f0eac4fbcbc46432ca5f0c6a4 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/7eb2667f0eac4fbcbc46432ca5f0c6a4 2023-03-30 16:55:19,073 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/6e193d869ce64607bd572ae93ff0197f 2023-03-30 16:55:19,077 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e3fe0ff55782e6f5d2f55c861f91d3cb/C of e3fe0ff55782e6f5d2f55c861f91d3cb into 7eb2667f0eac4fbcbc46432ca5f0c6a4(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:19,077 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:19,077 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb., storeName=e3fe0ff55782e6f5d2f55c861f91d3cb/C, priority=10, startTime=1680195318596; duration=0sec 2023-03-30 16:55:19,077 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:19,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:19,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60892 deadline: 1680195379104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:19,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195379129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:19,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195379151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:19,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60908 deadline: 1680195379152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:19,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195379152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:19,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195379434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:19,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195379458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:19,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195379458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:19,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60908 deadline: 1680195379459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,497 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/1fad2eb5eca449c2915b372cb05e7339 2023-03-30 16:55:19,501 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/828e1447ec224afe8f962ebea27626ef as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/828e1447ec224afe8f962ebea27626ef 2023-03-30 16:55:19,505 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/828e1447ec224afe8f962ebea27626ef, entries=150, sequenceid=250, filesize=25.3 K 2023-03-30 16:55:19,505 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/6e193d869ce64607bd572ae93ff0197f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/6e193d869ce64607bd572ae93ff0197f 2023-03-30 16:55:19,510 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/6e193d869ce64607bd572ae93ff0197f, entries=150, sequenceid=250, filesize=11.8 K 2023-03-30 16:55:19,511 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/1fad2eb5eca449c2915b372cb05e7339 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/1fad2eb5eca449c2915b372cb05e7339 2023-03-30 16:55:19,514 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/1fad2eb5eca449c2915b372cb05e7339, entries=150, sequenceid=250, filesize=11.8 K 2023-03-30 16:55:19,515 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for e3fe0ff55782e6f5d2f55c861f91d3cb in 888ms, sequenceid=250, compaction requested=false 2023-03-30 16:55:19,515 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:19,515 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:19,515 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:19,515 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:19,515 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,516 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:19,516 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:19,518 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:19,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:19,518 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:19,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:19,518 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:19,518 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:19,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:19,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:19,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:19,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:19,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2308c273[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:19,519 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:19,520 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:19,520 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:19,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:19,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:19,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:19,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:19,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:19,521 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,521 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:19,522 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,522 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:19,522 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:19,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:19,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:19,522 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:19,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:19,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:19,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:19,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:19,524 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:19,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:19,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:19,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:19,524 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:19,526 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:19,526 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:19,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:19,526 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:19,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:19,526 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:19,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:19,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:19,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:19,532 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,537 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:19,537 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:19,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:19,538 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,538 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:19,538 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:19,538 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:19,538 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,538 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:19,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:19,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:19,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:19,538 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:19,539 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:19,539 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:19,539 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:19,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:19,639 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:19,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:19,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:19,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:19,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:19,642 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:19,642 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:19,642 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:19,642 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:19,643 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:19,643 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:19,644 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:19,644 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:19,644 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,644 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:19,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:19,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:19,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:19,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:19,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:19,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:19,645 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:19,645 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:19,645 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:19,645 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:19,645 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:19,645 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:19,645 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:19,645 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-30 16:55:19,646 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:19,646 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:19,646 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:19,646 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:19,646 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:19,646 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:19,857 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/5b9af8919f75459ca85adde47a843652 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/5b9af8919f75459ca85adde47a843652 2023-03-30 16:55:19,861 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e3fe0ff55782e6f5d2f55c861f91d3cb/A of e3fe0ff55782e6f5d2f55c861f91d3cb into 5b9af8919f75459ca85adde47a843652(size=25.8 K), total size for store is 51.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-30 16:55:19,861 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:19,862 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb., storeName=e3fe0ff55782e6f5d2f55c861f91d3cb/A, priority=10, startTime=1680195318593; duration=1sec 2023-03-30 16:55:19,862 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:19,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:19,944 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:19,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:19,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60908 deadline: 1680195379967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:19,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195379968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:19,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195379972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:19,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:19,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195379974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:20,062 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330db7223a37f91481eae2be0775a2b800e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330db7223a37f91481eae2be0775a2b800e 2023-03-30 16:55:20,081 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:20,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195380074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:20,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:20,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195380075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:20,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:20,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195380079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:20,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:20,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195380283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:20,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:20,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195380284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:20,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:20,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195380284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:20,467 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=271, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/e4fa60e7e9a74d1f9dbcbab37b52d7a6 2023-03-30 16:55:20,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:20,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195380589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:20,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:20,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195380590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:20,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:20,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195380591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:20,878 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/44b902d025d9441185b6d86a04213cc7 2023-03-30 16:55:20,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:20,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60908 deadline: 1680195380979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:21,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195381093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:21,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195381095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:21,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195381097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:21,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60892 deadline: 1680195381126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,128 DEBUG [Thread-2165] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:55:21,289 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/e239bce646a44666b58ac67f8376583d 2023-03-30 16:55:21,293 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/e4fa60e7e9a74d1f9dbcbab37b52d7a6 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/e4fa60e7e9a74d1f9dbcbab37b52d7a6 2023-03-30 16:55:21,296 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/e4fa60e7e9a74d1f9dbcbab37b52d7a6, entries=150, sequenceid=271, filesize=25.4 K 2023-03-30 16:55:21,297 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/44b902d025d9441185b6d86a04213cc7 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/44b902d025d9441185b6d86a04213cc7 2023-03-30 16:55:21,301 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/44b902d025d9441185b6d86a04213cc7, entries=150, sequenceid=271, filesize=11.9 K 2023-03-30 16:55:21,302 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/e239bce646a44666b58ac67f8376583d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/e239bce646a44666b58ac67f8376583d 2023-03-30 16:55:21,305 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/e239bce646a44666b58ac67f8376583d, entries=150, sequenceid=271, filesize=11.9 K 2023-03-30 16:55:21,305 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for e3fe0ff55782e6f5d2f55c861f91d3cb in 1660ms, sequenceid=271, compaction requested=true 2023-03-30 16:55:21,306 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:21,306 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:21,306 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:21,306 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:21,306 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,306 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:21,306 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:21,307 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,307 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:21,308 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:21,308 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:21,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:21,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:21,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c1e793d[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:21,309 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:21,309 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:21,310 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:21,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:21,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:21,311 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:21,311 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:21,311 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,311 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:21,311 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:21,312 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,312 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:21,312 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:21,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:21,313 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:21,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:21,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:21,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:21,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:21,315 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:21,315 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:21,315 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:21,315 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:21,315 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:21,317 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:21,318 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:21,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:21,318 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:21,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:21,318 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:21,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:21,319 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:21,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:21,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,324 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,324 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:21,324 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:21,324 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,324 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:21,324 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:21,324 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:21,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:21,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:21,324 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:21,324 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:21,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:21,324 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:21,324 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:21,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:21,324 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:21,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:21,425 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:21,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:21,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:21,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:21,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:21,432 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:21,432 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:21,432 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:21,432 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:21,432 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:21,432 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:21,436 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,436 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:21,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:21,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:21,436 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:21,436 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:21,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:21,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:21,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:21,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:21,438 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:21,438 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:21,438 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:21,440 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:21,440 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:21,440 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:21,440 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:21,440 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-30 16:55:21,440 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:21,440 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:21,440 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:21,440 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:21,440 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:21,440 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:21,860 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303303104cbe9b452463bb8fab71d84a4c7d6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303104cbe9b452463bb8fab71d84a4c7d6 2023-03-30 16:55:21,873 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=286, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/989f0e0a80384ac587429e29316b6f80 2023-03-30 16:55:22,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:22,108 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:22,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:22,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195382138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:22,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195382139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:22,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195382145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:22,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195382249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:22,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195382251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:22,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195382251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,284 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/43d3ba2e54e340fea04060f55789d24f 2023-03-30 16:55:22,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:22,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195382453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:22,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195382455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:22,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195382456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,693 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/6e09bf9108434157942f20d78b3b48cb 2023-03-30 16:55:22,697 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/989f0e0a80384ac587429e29316b6f80 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/989f0e0a80384ac587429e29316b6f80 2023-03-30 16:55:22,700 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/989f0e0a80384ac587429e29316b6f80, entries=150, sequenceid=286, filesize=25.4 K 2023-03-30 16:55:22,701 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/43d3ba2e54e340fea04060f55789d24f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/43d3ba2e54e340fea04060f55789d24f 2023-03-30 16:55:22,706 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/43d3ba2e54e340fea04060f55789d24f, entries=150, sequenceid=286, filesize=11.9 K 2023-03-30 16:55:22,706 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/6e09bf9108434157942f20d78b3b48cb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/6e09bf9108434157942f20d78b3b48cb 2023-03-30 16:55:22,710 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/6e09bf9108434157942f20d78b3b48cb, entries=150, sequenceid=286, filesize=11.9 K 2023-03-30 16:55:22,711 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for e3fe0ff55782e6f5d2f55c861f91d3cb in 1271ms, sequenceid=286, compaction requested=true 2023-03-30 16:55:22,711 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:22,711 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:22,711 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:22,711 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:22,711 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,711 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:22,711 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:22,714 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,714 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:22,714 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,714 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:22,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:22,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:22,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:22,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b6807b0[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:22,715 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:22,716 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,717 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:22,717 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,717 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:22,718 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:22,718 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:22,718 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,718 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:22,721 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:22,721 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:22,721 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:22,721 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:22,721 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:22,723 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,723 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:22,723 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:22,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:22,723 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:22,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:22,726 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,729 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:22,729 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,729 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:22,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:22,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:22,729 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:22,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:22,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:22,729 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:22,730 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:22,730 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,730 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,730 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,730 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:22,730 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:22,730 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:22,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:22,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:22,763 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-30 16:55:22,764 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:22,764 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:22,764 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:22,764 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:22,764 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:22,764 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:22,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:22,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195382785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:22,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195382786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:22,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195382787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,830 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:22,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:22,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:22,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:22,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:22,833 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:22,833 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:22,833 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:22,833 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:22,834 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,834 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:22,835 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:22,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:22,835 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:22,835 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,835 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:22,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:22,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:22,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:22,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:22,836 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:22,836 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:22,836 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:22,836 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:22,836 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:22,836 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:22,836 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:22,836 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:22,836 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:22,837 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:22,837 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:22,837 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,837 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:22,837 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:22,838 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,838 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:22,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:22,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:22,838 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,838 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:22,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:22,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:22,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:22,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:22,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10a53611[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:22,840 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:22,840 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,841 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,841 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:22,841 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:22,841 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:22,841 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,841 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:22,844 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:22,844 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:22,844 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,844 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:22,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:22,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:22,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:22,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:22,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:22,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:22,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:22,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:22,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:22,846 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:22,846 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:22,846 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:22,846 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:22,846 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:22,848 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,848 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,848 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:22,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:22,848 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:22,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:22,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:22,849 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:22,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:22,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,851 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,851 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:22,851 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:22,851 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,851 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:22,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:22,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:22,851 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:22,851 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,851 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:22,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:22,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:22,852 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:22,852 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:22,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:22,852 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,852 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:22,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:22,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:22,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195382893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:22,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195382894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:22,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195382894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,952 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:22,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:22,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:22,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:22,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:22,955 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:22,955 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:22,955 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:22,955 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:22,956 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,956 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:22,957 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:22,957 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:22,957 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,957 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:22,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:22,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:22,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:22,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:22,958 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:22,958 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:22,959 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:22,959 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:22,959 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:22,959 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:22,959 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,959 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:22,959 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:22,961 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,961 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:22,961 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,961 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:22,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:22,963 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:22,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755b652c[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:22,963 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,964 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,964 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:22,964 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:22,964 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:22,964 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,964 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:22,965 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:22,965 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:22,965 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,965 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:22,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:22,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:22,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:22,968 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:22,968 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:22,968 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:22,968 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:22,968 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:22,969 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,969 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:22,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,969 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:22,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:22,970 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:22,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:22,971 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:22,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:22,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:22,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:22,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:22,977 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:22,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:22,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:22,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:22,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:22,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:22,978 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:22,978 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,978 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:22,978 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:22,978 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:22,978 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:22,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:22,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:22,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:23,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:23,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60908 deadline: 1680195382995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,001 DEBUG [Thread-2167] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4183 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:55:23,078 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:23,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:23,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:23,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:23,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:23,082 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:23,082 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:23,082 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:23,082 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:23,082 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,082 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,083 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,084 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:23,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:23,084 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,084 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:23,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:23,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:23,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:23,085 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:23,085 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:23,085 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:23,085 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:23,085 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:23,085 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:23,085 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:23,085 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:23,085 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:23,086 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:23,086 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:23,086 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,086 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:23,086 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:23,087 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:23,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ae32d2b[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:23,089 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:23,089 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,089 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,090 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,090 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:23,090 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:23,090 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:23,090 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:23,092 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,092 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:23,092 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:23,092 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:23,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:23,094 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:23,094 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:23,094 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:23,094 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:23,094 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:23,096 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,096 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,096 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:23,097 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,098 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,101 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,101 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,101 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:23,101 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:23,101 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,101 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:23,101 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,101 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:23,101 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:23,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:23,101 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,102 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:23,102 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:23,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:23,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:23,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195383099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:23,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195383099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:23,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195383100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,177 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303302f8bf8d119974ae3a4657ce996ff5bf3 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303302f8bf8d119974ae3a4657ce996ff5bf3 2023-03-30 16:55:23,183 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=305, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/c465cf722361474ab5b9f91775b93104 2023-03-30 16:55:23,202 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:23,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:23,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:23,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:23,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:23,216 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:23,216 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:23,216 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:23,216 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:23,216 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,217 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,218 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,218 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:23,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:23,218 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,218 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:23,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:23,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:23,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:23,219 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:23,219 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:23,219 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:23,221 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:23,221 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:23,221 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:23,221 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:23,221 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:23,222 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:23,222 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:23,222 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:23,222 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,222 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:23,222 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:23,225 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,225 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,225 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,225 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:23,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:23,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@311dfc40[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:23,227 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:23,227 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,228 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,228 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:23,228 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:23,228 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,228 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:23,228 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:23,230 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,230 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:23,230 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:23,230 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:23,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:23,232 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:23,233 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:23,233 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:23,233 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:23,233 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:23,234 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,234 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,234 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:23,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,235 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,237 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,239 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:23,239 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:23,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:23,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:23,239 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,240 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,240 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:23,240 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:23,240 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:23,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:23,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:23,240 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:23,240 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,240 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:23,240 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,240 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,340 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:23,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:23,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:23,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:23,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:23,344 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:23,344 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:23,344 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:23,344 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:23,344 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,344 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,345 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,346 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,346 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:23,347 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:23,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:23,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:23,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:23,348 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:23,348 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:23,348 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:23,348 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:23,348 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:23,348 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:23,348 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:23,348 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:23,348 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:23,348 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:23,348 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:23,349 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,349 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:23,349 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:23,350 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,350 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,350 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,350 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:23,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8b141cc[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:23,351 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:23,352 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,353 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,353 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:23,353 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:23,353 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:23,353 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,353 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:23,355 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,355 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,355 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:23,355 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:23,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:23,358 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:23,358 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:23,358 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:23,358 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:23,358 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:23,359 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,359 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:23,359 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:23,360 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,361 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,365 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,365 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,365 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,365 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:23,365 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:23,365 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,365 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:23,365 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:23,365 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,365 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:23,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:23,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:23,365 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:23,365 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:23,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:23,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:23,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:23,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195383404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:23,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195383406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:23,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195383406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,466 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:23,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:23,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:23,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:23,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:23,469 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:23,469 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:23,469 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:23,469 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:23,469 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,469 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,470 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:23,470 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:23,471 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,471 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:23,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:23,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:23,472 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:23,472 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:23,472 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:23,472 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:23,472 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:23,472 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:23,472 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:23,472 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:23,472 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:23,472 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:23,472 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:23,472 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,472 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:23,472 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:23,477 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,477 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,477 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,477 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:23,478 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:23,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@266d37f3[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:23,478 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,479 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,480 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:23,480 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:23,480 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:23,480 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,480 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:23,481 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:23,481 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:23,481 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,481 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:23,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:23,483 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:23,483 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:23,483 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:23,483 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:23,483 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:23,485 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,485 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,485 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,485 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,486 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,489 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,489 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,489 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:23,489 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:23,489 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:23,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:23,489 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:23,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:23,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:23,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:23,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:23,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:23,489 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,490 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:23,490 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:23,490 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,490 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,490 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,590 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:23,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:23,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:23,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:23,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:23,593 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:23,593 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:23,593 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:23,593 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:23,594 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,594 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,595 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:23,595 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:23,595 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,595 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:23,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,597 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:23,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:23,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:23,597 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:23,597 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:23,597 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:23,597 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:23,597 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:23,597 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:23,597 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:23,597 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:23,597 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:23,598 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:23,598 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:23,598 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,598 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:23,598 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:23,599 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,599 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,599 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,599 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:23,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:23,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25e4dc60[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:23,601 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:23,601 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,602 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,602 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:23,602 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,602 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:23,602 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:23,602 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:23,604 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,604 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,604 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:23,605 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:23,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:23,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:23,606 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:23,607 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:23,607 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:23,607 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:23,607 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:23,608 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,608 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,608 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:23,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:23,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:23,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,610 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,611 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,613 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,613 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,613 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:23,613 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:23,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:23,613 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,613 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:23,613 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:23,613 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:23,613 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:23,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:23,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:23,613 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,613 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,613 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,613 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/4c6ce09a96ee49a5952fc774d22c8cb4 2023-03-30 16:55:23,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:23,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:23,713 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:23,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:23,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:23,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:23,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:23,719 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:23,719 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:23,719 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:23,719 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:23,719 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,719 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,720 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,720 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:23,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:23,721 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,721 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:23,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:23,722 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:23,722 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:23,722 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:23,728 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:23,728 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:23,728 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:23,728 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:23,728 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:23,728 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:23,728 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:23,728 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:23,728 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,728 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:23,728 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:23,734 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,734 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,735 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,735 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:23,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:23,736 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:23,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b45cb60[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:23,736 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,738 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,738 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:23,738 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:23,738 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:23,738 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,738 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:23,739 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,739 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:23,739 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:23,739 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:23,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:23,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:23,742 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:23,742 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:23,742 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:23,742 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:23,742 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:23,743 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,744 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,744 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,744 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:23,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,745 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:23,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,748 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:23,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:23,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:23,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:23,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:23,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:23,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:23,750 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:23,750 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:23,748 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,750 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,750 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,750 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,750 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:23,850 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:23,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:23,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:23,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:23,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:23,854 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:23,854 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:23,854 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:23,854 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:23,854 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,854 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,856 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,856 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:23,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:23,856 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,856 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:23,857 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:23,858 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:23,858 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:23,858 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:23,858 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:23,858 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:23,858 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:23,858 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:23,858 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:23,858 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:23,858 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:23,858 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:23,858 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:23,859 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,859 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:23,859 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:23,860 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,860 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,860 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,860 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:23,862 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:23,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79784ae5[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:23,862 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,863 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,863 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:23,863 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:23,863 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:23,863 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,863 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:23,868 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:23,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:23,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:23,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:23,871 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:23,871 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:23,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:23,871 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:23,871 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:23,871 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:23,872 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,872 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,872 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:23,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,873 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:23,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:23,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,874 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,876 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,876 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,876 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,876 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,876 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,876 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:23,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:23,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:23,876 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:23,876 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:23,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:23,876 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:23,876 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:23,876 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:23,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:23,876 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:23,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:23,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:23,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195383917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:23,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195383917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:23,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195383917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,976 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:23,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:23,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:23,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:23,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:23,980 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:23,980 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:23,980 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:23,980 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:23,980 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,980 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,981 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,981 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:23,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:23,982 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,982 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:23,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:23,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:23,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:23,983 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:23,983 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:23,983 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:23,983 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:23,983 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:23,983 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:23,983 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:23,983 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:23,983 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:23,983 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:23,983 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:23,983 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,983 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:23,983 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:23,986 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,986 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,987 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,987 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:23,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36653896[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:23,988 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:23,988 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,989 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:23,990 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:23,990 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,990 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:23,990 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:23,990 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:23,991 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:23,991 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,991 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,991 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:23,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:23,993 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:23,993 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:23,993 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:23,993 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:23,993 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:23,994 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,994 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:23,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:23,995 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:23,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:23,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:23,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:23,995 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:23,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:23,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,996 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:23,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:23,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:23,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:24,000 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:24,000 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:24,000 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:24,000 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:24,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:24,000 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:24,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:24,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:24,000 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:24,000 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:24,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:24,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:24,000 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:24,000 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:24,001 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:24,000 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:24,001 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:24,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:24,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:24,024 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/0ac8abf09bc84a94bdcc8a5cd9644976 2023-03-30 16:55:24,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/c465cf722361474ab5b9f91775b93104 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c465cf722361474ab5b9f91775b93104 2023-03-30 16:55:24,031 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c465cf722361474ab5b9f91775b93104, entries=200, sequenceid=305, filesize=32.3 K 2023-03-30 16:55:24,032 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/4c6ce09a96ee49a5952fc774d22c8cb4 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/4c6ce09a96ee49a5952fc774d22c8cb4 2023-03-30 16:55:24,035 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/4c6ce09a96ee49a5952fc774d22c8cb4, entries=150, sequenceid=305, filesize=11.9 K 2023-03-30 16:55:24,036 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/0ac8abf09bc84a94bdcc8a5cd9644976 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/0ac8abf09bc84a94bdcc8a5cd9644976 2023-03-30 16:55:24,039 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/0ac8abf09bc84a94bdcc8a5cd9644976, entries=150, sequenceid=305, filesize=11.9 K 2023-03-30 16:55:24,040 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for e3fe0ff55782e6f5d2f55c861f91d3cb in 1277ms, sequenceid=305, compaction requested=true 2023-03-30 16:55:24,040 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:24,040 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:24,040 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:24,040 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 16:55:24,040 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 16:55:24,040 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:24,041 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137516 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 16:55:24,041 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61176 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 16:55:24,042 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): e3fe0ff55782e6f5d2f55c861f91d3cb/A is initiating minor compaction (all files) 2023-03-30 16:55:24,042 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): e3fe0ff55782e6f5d2f55c861f91d3cb/B is initiating minor compaction (all files) 2023-03-30 16:55:24,042 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e3fe0ff55782e6f5d2f55c861f91d3cb/A in TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:24,042 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3fe0ff55782e6f5d2f55c861f91d3cb/B in TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:24,042 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/5b9af8919f75459ca85adde47a843652, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/828e1447ec224afe8f962ebea27626ef, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/e4fa60e7e9a74d1f9dbcbab37b52d7a6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/989f0e0a80384ac587429e29316b6f80, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c465cf722361474ab5b9f91775b93104] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp, totalSize=134.3 K 2023-03-30 16:55:24,042 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/381daa52fdc242b59ccda3fe537cf08f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/6e193d869ce64607bd572ae93ff0197f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/44b902d025d9441185b6d86a04213cc7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/43d3ba2e54e340fea04060f55789d24f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/4c6ce09a96ee49a5952fc774d22c8cb4] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp, totalSize=59.7 K 2023-03-30 16:55:24,042 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 5b9af8919f75459ca85adde47a843652, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680195316972 2023-03-30 16:55:24,042 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 381daa52fdc242b59ccda3fe537cf08f, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680195316972 2023-03-30 16:55:24,042 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 828e1447ec224afe8f962ebea27626ef, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1680195318142 2023-03-30 16:55:24,043 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 6e193d869ce64607bd572ae93ff0197f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1680195318142 2023-03-30 16:55:24,043 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting e4fa60e7e9a74d1f9dbcbab37b52d7a6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1680195318807 2023-03-30 16:55:24,043 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 44b902d025d9441185b6d86a04213cc7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1680195318807 2023-03-30 16:55:24,043 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 989f0e0a80384ac587429e29316b6f80, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680195319957 2023-03-30 16:55:24,043 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 43d3ba2e54e340fea04060f55789d24f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680195319957 2023-03-30 16:55:24,044 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting c465cf722361474ab5b9f91775b93104, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1680195322131 2023-03-30 16:55:24,044 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 4c6ce09a96ee49a5952fc774d22c8cb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1680195322131 2023-03-30 16:55:24,055 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): e3fe0ff55782e6f5d2f55c861f91d3cb#B#compaction#335 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:24,067 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): e3fe0ff55782e6f5d2f55c861f91d3cb#A#compaction#336 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:24,090 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/cb520ef119aa4c15a747700f66348773 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/cb520ef119aa4c15a747700f66348773 2023-03-30 16:55:24,095 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e3fe0ff55782e6f5d2f55c861f91d3cb/B of e3fe0ff55782e6f5d2f55c861f91d3cb into cb520ef119aa4c15a747700f66348773(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:24,095 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:24,095 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb., storeName=e3fe0ff55782e6f5d2f55c861f91d3cb/B, priority=11, startTime=1680195324040; duration=0sec 2023-03-30 16:55:24,095 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:24,095 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 16:55:24,097 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61176 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 16:55:24,098 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): e3fe0ff55782e6f5d2f55c861f91d3cb/C is initiating minor compaction (all files) 2023-03-30 16:55:24,098 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3fe0ff55782e6f5d2f55c861f91d3cb/C in TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:24,098 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/7eb2667f0eac4fbcbc46432ca5f0c6a4, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/1fad2eb5eca449c2915b372cb05e7339, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/e239bce646a44666b58ac67f8376583d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/6e09bf9108434157942f20d78b3b48cb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/0ac8abf09bc84a94bdcc8a5cd9644976] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp, totalSize=59.7 K 2023-03-30 16:55:24,098 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 7eb2667f0eac4fbcbc46432ca5f0c6a4, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680195316972 2023-03-30 16:55:24,098 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 1fad2eb5eca449c2915b372cb05e7339, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1680195318142 2023-03-30 16:55:24,099 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting e239bce646a44666b58ac67f8376583d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1680195318807 2023-03-30 16:55:24,099 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 6e09bf9108434157942f20d78b3b48cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1680195319957 2023-03-30 16:55:24,101 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 0ac8abf09bc84a94bdcc8a5cd9644976, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1680195322131 2023-03-30 16:55:24,101 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:24,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:24,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:24,114 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): e3fe0ff55782e6f5d2f55c861f91d3cb#C#compaction#337 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:24,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:24,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:24,115 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:24,115 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:24,115 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:24,115 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:24,115 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:24,115 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:24,116 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:24,116 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:24,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:24,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:24,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:24,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:24,118 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:24,118 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:24,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:24,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:24,124 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:24,124 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:24,124 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:24,128 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:24,128 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:24,128 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:24,128 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. started... 2023-03-30 16:55:24,128 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-30 16:55:24,129 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:24,129 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:24,129 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:24,129 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:24,129 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:24,129 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:24,140 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/d9f6b6e852ae4de7af5740698e765c1e as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/d9f6b6e852ae4de7af5740698e765c1e 2023-03-30 16:55:24,147 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e3fe0ff55782e6f5d2f55c861f91d3cb/C of e3fe0ff55782e6f5d2f55c861f91d3cb into d9f6b6e852ae4de7af5740698e765c1e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:24,147 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:24,147 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb., storeName=e3fe0ff55782e6f5d2f55c861f91d3cb/C, priority=11, startTime=1680195324040; duration=0sec 2023-03-30 16:55:24,147 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:24,529 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/78a1cf7a0b024023be50eb5c57e9d792 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/78a1cf7a0b024023be50eb5c57e9d792 2023-03-30 16:55:24,534 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e3fe0ff55782e6f5d2f55c861f91d3cb/A of e3fe0ff55782e6f5d2f55c861f91d3cb into 78a1cf7a0b024023be50eb5c57e9d792(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:24,534 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:24,534 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb., storeName=e3fe0ff55782e6f5d2f55c861f91d3cb/A, priority=11, startTime=1680195324040; duration=0sec 2023-03-30 16:55:24,534 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:24,558 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303303e044154fbcd46949396628847441017 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303e044154fbcd46949396628847441017 2023-03-30 16:55:24,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:24,922 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. as already flushing 2023-03-30 16:55:24,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:24,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195384942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:24,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:24,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195384943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:24,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:24,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195384945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:24,964 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=319, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/051f352a0c4d40e38246e25a59ed8e62 2023-03-30 16:55:25,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:25,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:25,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195385047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195385048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:25,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195385050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:25,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60892 deadline: 1680195385129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,136 DEBUG [Thread-2165] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:55:25,244 DEBUG [Thread-2178] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ea43962 to 127.0.0.1:58435 2023-03-30 16:55:25,244 DEBUG [Thread-2178] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:55:25,248 DEBUG [Thread-2172] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x722b2407 to 127.0.0.1:58435 2023-03-30 16:55:25,248 DEBUG [Thread-2172] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:55:25,249 DEBUG [Thread-2174] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x066bf91c to 127.0.0.1:58435 2023-03-30 16:55:25,249 DEBUG [Thread-2174] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:55:25,250 DEBUG [Thread-2180] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2abd7961 to 127.0.0.1:58435 2023-03-30 16:55:25,250 DEBUG [Thread-2180] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:55:25,253 DEBUG [Thread-2176] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6846dddb to 127.0.0.1:58435 2023-03-30 16:55:25,254 DEBUG [Thread-2176] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:55:25,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:25,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195385255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:25,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:25,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195385256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195385256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,374 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/dc5f37e0301a4bd8bef24a79fbb5db30 2023-03-30 16:55:25,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:25,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60926 deadline: 1680195385557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,559 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:25,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:25,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60884 deadline: 1680195385559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60866 deadline: 1680195385559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,782 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/902fe6e43d9b453a9ff87f37886caf37 2023-03-30 16:55:25,785 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/051f352a0c4d40e38246e25a59ed8e62 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/051f352a0c4d40e38246e25a59ed8e62 2023-03-30 16:55:25,788 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/051f352a0c4d40e38246e25a59ed8e62, entries=150, sequenceid=319, filesize=25.4 K 2023-03-30 16:55:25,789 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/dc5f37e0301a4bd8bef24a79fbb5db30 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/dc5f37e0301a4bd8bef24a79fbb5db30 2023-03-30 16:55:25,791 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/dc5f37e0301a4bd8bef24a79fbb5db30, entries=150, sequenceid=319, filesize=11.9 K 2023-03-30 16:55:25,792 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/902fe6e43d9b453a9ff87f37886caf37 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/902fe6e43d9b453a9ff87f37886caf37 2023-03-30 16:55:25,795 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/902fe6e43d9b453a9ff87f37886caf37, entries=150, sequenceid=319, filesize=11.9 K 2023-03-30 16:55:25,795 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for e3fe0ff55782e6f5d2f55c861f91d3cb in 1667ms, sequenceid=319, compaction requested=false 2023-03-30 16:55:25,795 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:25,795 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:25,795 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:25,795 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:25,795 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,795 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:25,795 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:25,797 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:25,797 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:25,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:25,797 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:25,797 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:25,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:25,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:25,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:25,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:25,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:25,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35e1f5e6[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:25,798 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:25,798 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:25,800 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:25,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:25,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:25,800 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:25,800 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:25,800 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:25,800 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,800 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:25,801 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:25,801 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,801 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:25,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:25,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:25,801 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:25,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:25,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:25,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:25,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:25,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:25,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:25,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:25,803 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:25,803 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:25,803 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:25,803 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:25,803 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:25,805 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:25,805 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:25,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:25,805 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:25,805 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:25,806 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:25,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:25,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:25,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:25,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:25,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:25,808 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:25,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:25,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:25,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:25,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:25,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:25,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:25,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:25,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:25,808 DEBUG [Thread-2171] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:25,808 DEBUG [Thread-2171] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:25,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:25,808 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:25,908 DEBUG [Thread-2171] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:25,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:26,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:26,062 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-30 16:55:26,062 DEBUG [Thread-2169] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5b7d4a51 to 127.0.0.1:58435 2023-03-30 16:55:26,062 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:26,062 DEBUG [Thread-2169] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:55:26,062 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:26,062 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:26,062 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:26,062 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:26,062 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:26,063 DEBUG [Thread-2161] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41798291 to 127.0.0.1:58435 2023-03-30 16:55:26,063 DEBUG [Thread-2161] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:55:26,064 DEBUG [Thread-2163] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x118e131f to 127.0.0.1:58435 2023-03-30 16:55:26,064 DEBUG [Thread-2163] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:55:26,473 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033090352b4a37fa4f0183db06c0872666fd to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033090352b4a37fa4f0183db06c0872666fd 2023-03-30 16:55:26,877 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=341, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/e9a22b4bbdfc4271a0388ba1034ab596 2023-03-30 16:55:27,036 DEBUG [Thread-2167] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6e7e18ef to 127.0.0.1:58435 2023-03-30 16:55:27,036 DEBUG [Thread-2167] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:55:27,286 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/bfdb5d162c654278848f514b86662524 2023-03-30 16:55:27,695 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/9e27e7870e554341b929758944178515 2023-03-30 16:55:27,698 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/e9a22b4bbdfc4271a0388ba1034ab596 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/e9a22b4bbdfc4271a0388ba1034ab596 2023-03-30 16:55:27,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/e9a22b4bbdfc4271a0388ba1034ab596, entries=150, sequenceid=341, filesize=25.4 K 2023-03-30 16:55:27,701 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/bfdb5d162c654278848f514b86662524 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/bfdb5d162c654278848f514b86662524 2023-03-30 16:55:27,704 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/bfdb5d162c654278848f514b86662524, entries=150, sequenceid=341, filesize=11.9 K 2023-03-30 16:55:27,705 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/9e27e7870e554341b929758944178515 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/9e27e7870e554341b929758944178515 2023-03-30 16:55:27,708 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/9e27e7870e554341b929758944178515, entries=150, sequenceid=341, filesize=11.9 K 2023-03-30 16:55:27,708 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=20.13 KB/20610 for e3fe0ff55782e6f5d2f55c861f91d3cb in 1646ms, sequenceid=341, compaction requested=true 2023-03-30 16:55:27,708 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:27,708 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:27,708 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:27,708 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:55:27,708 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:55:27,708 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:27,709 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78854 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:55:27,709 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): e3fe0ff55782e6f5d2f55c861f91d3cb/A is initiating minor compaction (all files) 2023-03-30 16:55:27,709 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:55:27,709 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e3fe0ff55782e6f5d2f55c861f91d3cb/A in TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:27,709 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): e3fe0ff55782e6f5d2f55c861f91d3cb/B is initiating minor compaction (all files) 2023-03-30 16:55:27,710 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/78a1cf7a0b024023be50eb5c57e9d792, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/051f352a0c4d40e38246e25a59ed8e62, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/e9a22b4bbdfc4271a0388ba1034ab596] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp, totalSize=77.0 K 2023-03-30 16:55:27,710 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3fe0ff55782e6f5d2f55c861f91d3cb/B in TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:27,710 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/cb520ef119aa4c15a747700f66348773, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/dc5f37e0301a4bd8bef24a79fbb5db30, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/bfdb5d162c654278848f514b86662524] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp, totalSize=36.4 K 2023-03-30 16:55:27,710 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 78a1cf7a0b024023be50eb5c57e9d792, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1680195322131 2023-03-30 16:55:27,710 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting cb520ef119aa4c15a747700f66348773, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1680195322131 2023-03-30 16:55:27,710 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 051f352a0c4d40e38246e25a59ed8e62, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1680195322777 2023-03-30 16:55:27,710 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting dc5f37e0301a4bd8bef24a79fbb5db30, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1680195322777 2023-03-30 16:55:27,710 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting e9a22b4bbdfc4271a0388ba1034ab596, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1680195324941 2023-03-30 16:55:27,710 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting bfdb5d162c654278848f514b86662524, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1680195324941 2023-03-30 16:55:27,721 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): e3fe0ff55782e6f5d2f55c861f91d3cb#B#compaction#345 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:27,723 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): e3fe0ff55782e6f5d2f55c861f91d3cb#A#compaction#344 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:28,131 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/a64f437df89a4a1e8bea181b55b3c180 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/a64f437df89a4a1e8bea181b55b3c180 2023-03-30 16:55:28,135 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3fe0ff55782e6f5d2f55c861f91d3cb/B of e3fe0ff55782e6f5d2f55c861f91d3cb into a64f437df89a4a1e8bea181b55b3c180(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:28,135 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:28,135 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb., storeName=e3fe0ff55782e6f5d2f55c861f91d3cb/B, priority=13, startTime=1680195327708; duration=0sec 2023-03-30 16:55:28,135 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:28,135 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:55:28,136 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:55:28,136 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): e3fe0ff55782e6f5d2f55c861f91d3cb/C is initiating minor compaction (all files) 2023-03-30 16:55:28,136 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e3fe0ff55782e6f5d2f55c861f91d3cb/C in TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:28,136 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/d9f6b6e852ae4de7af5740698e765c1e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/902fe6e43d9b453a9ff87f37886caf37, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/9e27e7870e554341b929758944178515] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp, totalSize=36.4 K 2023-03-30 16:55:28,137 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting d9f6b6e852ae4de7af5740698e765c1e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1680195322131 2023-03-30 16:55:28,137 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 902fe6e43d9b453a9ff87f37886caf37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1680195322777 2023-03-30 16:55:28,137 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 9e27e7870e554341b929758944178515, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1680195324941 2023-03-30 16:55:28,144 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): e3fe0ff55782e6f5d2f55c861f91d3cb#C#compaction#346 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:28,543 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/cd432fceef1844b3afff94011b7ee160 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/cd432fceef1844b3afff94011b7ee160 2023-03-30 16:55:28,547 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3fe0ff55782e6f5d2f55c861f91d3cb/A of e3fe0ff55782e6f5d2f55c861f91d3cb into cd432fceef1844b3afff94011b7ee160(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:28,547 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:28,547 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb., storeName=e3fe0ff55782e6f5d2f55c861f91d3cb/A, priority=13, startTime=1680195327708; duration=0sec 2023-03-30 16:55:28,547 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:28,550 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/77718c35daca4a7cbe28f3f8ea978b97 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/77718c35daca4a7cbe28f3f8ea978b97 2023-03-30 16:55:28,554 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e3fe0ff55782e6f5d2f55c861f91d3cb/C of e3fe0ff55782e6f5d2f55c861f91d3cb into 77718c35daca4a7cbe28f3f8ea978b97(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:28,554 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:28,554 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb., storeName=e3fe0ff55782e6f5d2f55c861f91d3cb/C, priority=13, startTime=1680195327708; duration=0sec 2023-03-30 16:55:28,554 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:35,154 DEBUG [Thread-2165] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2deaf1fa to 127.0.0.1:58435 2023-03-30 16:55:35,154 DEBUG [Thread-2165] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:55:35,154 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-30 16:55:35,154 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 63 2023-03-30 16:55:35,154 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 65 2023-03-30 16:55:35,154 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 36 2023-03-30 16:55:35,154 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 41 2023-03-30 16:55:35,154 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 64 2023-03-30 16:55:35,155 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-30 16:55:35,155 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-30 16:55:35,155 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(402): scanned 1893 2023-03-30 16:55:35,155 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(403): verified 5675 rows 2023-03-30 16:55:35,155 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(402): scanned 1891 2023-03-30 16:55:35,155 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(403): verified 5672 rows 2023-03-30 16:55:35,155 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(402): scanned 1881 2023-03-30 16:55:35,155 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(403): verified 5642 rows 2023-03-30 16:55:35,155 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(402): scanned 1885 2023-03-30 16:55:35,155 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(403): verified 5654 rows 2023-03-30 16:55:35,155 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(402): scanned 1898 2023-03-30 16:55:35,155 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(403): verified 5694 rows 2023-03-30 16:55:35,155 INFO [Listener at localhost/38075] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-30 16:55:35,155 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6038da2f to 127.0.0.1:58435 2023-03-30 16:55:35,155 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:55:35,157 INFO [Listener at localhost/38075] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-30 16:55:35,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-30 16:55:35,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-30 16:55:35,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-30 16:55:35,161 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195335161"}]},"ts":"1680195335161"} 2023-03-30 16:55:35,162 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-30 16:55:35,164 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-30 16:55:35,164 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e3fe0ff55782e6f5d2f55c861f91d3cb, UNASSIGN}] 2023-03-30 16:55:35,166 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e3fe0ff55782e6f5d2f55c861f91d3cb, UNASSIGN 2023-03-30 16:55:35,166 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=e3fe0ff55782e6f5d2f55c861f91d3cb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:35,166 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195335166"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680195335166"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680195335166"}]},"ts":"1680195335166"} 2023-03-30 16:55:35,167 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285}] 2023-03-30 16:55:35,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-30 16:55:35,318 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:35,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e3fe0ff55782e6f5d2f55c861f91d3cb, disabling compactions & flushes 2023-03-30 16:55:35,319 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:35,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:35,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. after waiting 0 ms 2023-03-30 16:55:35,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:35,319 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing e3fe0ff55782e6f5d2f55c861f91d3cb 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-30 16:55:35,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=A 2023-03-30 16:55:35,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:35,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=B 2023-03-30 16:55:35,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:35,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e3fe0ff55782e6f5d2f55c861f91d3cb, store=C 2023-03-30 16:55:35,319 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:35,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-30 16:55:35,640 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-30 16:55:35,663 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c202c30c81c44d6cb028459aae08eefe, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/406f8e923ce14c1d847d4ca94e6b3f40, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/45213b0b274d465091ddc7c78f8f598f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/f3e258bc59f64f31b9c707026b0392ff, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/73564254ccd94243b88f67dcd35228fb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/6cf862c234024c48b523fe502089e003, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/bd3ae11b9e4345e39eb7d7423592736b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/2dd0e65e8a08423387a1b84c49cbf021, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/6b380132a6874711a5cede0698af0058, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/b2500d6cd3634edaa8a8165057876348, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/a24a78ec3b404186aee26b342d86f53e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c13098f4a81e475f9a8fc52e24055c0f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c72a3ae0dc9e4ce38f1f87e13fe182df, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c994cbd7f0eb400ca2f9691e099759e0, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/3d4e10d342b2442d9f59b81d2b91d423, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/619c0dbe2aae48f591ab41725016137d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/5b9af8919f75459ca85adde47a843652, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/828e1447ec224afe8f962ebea27626ef, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/e4fa60e7e9a74d1f9dbcbab37b52d7a6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/989f0e0a80384ac587429e29316b6f80, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c465cf722361474ab5b9f91775b93104, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/78a1cf7a0b024023be50eb5c57e9d792, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/051f352a0c4d40e38246e25a59ed8e62, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/e9a22b4bbdfc4271a0388ba1034ab596] to archive 2023-03-30 16:55:35,664 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 16:55:35,666 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c202c30c81c44d6cb028459aae08eefe to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c202c30c81c44d6cb028459aae08eefe 2023-03-30 16:55:35,667 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/406f8e923ce14c1d847d4ca94e6b3f40 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/406f8e923ce14c1d847d4ca94e6b3f40 2023-03-30 16:55:35,668 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/45213b0b274d465091ddc7c78f8f598f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/45213b0b274d465091ddc7c78f8f598f 2023-03-30 16:55:35,669 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/f3e258bc59f64f31b9c707026b0392ff to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/f3e258bc59f64f31b9c707026b0392ff 2023-03-30 16:55:35,670 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/73564254ccd94243b88f67dcd35228fb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/73564254ccd94243b88f67dcd35228fb 2023-03-30 16:55:35,671 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/6cf862c234024c48b523fe502089e003 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/6cf862c234024c48b523fe502089e003 2023-03-30 16:55:35,672 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/bd3ae11b9e4345e39eb7d7423592736b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/bd3ae11b9e4345e39eb7d7423592736b 2023-03-30 16:55:35,674 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/2dd0e65e8a08423387a1b84c49cbf021 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/2dd0e65e8a08423387a1b84c49cbf021 2023-03-30 16:55:35,675 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/6b380132a6874711a5cede0698af0058 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/6b380132a6874711a5cede0698af0058 2023-03-30 16:55:35,676 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/b2500d6cd3634edaa8a8165057876348 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/b2500d6cd3634edaa8a8165057876348 2023-03-30 16:55:35,677 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/a24a78ec3b404186aee26b342d86f53e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/a24a78ec3b404186aee26b342d86f53e 2023-03-30 16:55:35,678 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c13098f4a81e475f9a8fc52e24055c0f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c13098f4a81e475f9a8fc52e24055c0f 2023-03-30 16:55:35,679 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c72a3ae0dc9e4ce38f1f87e13fe182df to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c72a3ae0dc9e4ce38f1f87e13fe182df 2023-03-30 16:55:35,680 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c994cbd7f0eb400ca2f9691e099759e0 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c994cbd7f0eb400ca2f9691e099759e0 2023-03-30 16:55:35,681 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/3d4e10d342b2442d9f59b81d2b91d423 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/3d4e10d342b2442d9f59b81d2b91d423 2023-03-30 16:55:35,682 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/619c0dbe2aae48f591ab41725016137d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/619c0dbe2aae48f591ab41725016137d 2023-03-30 16:55:35,683 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/5b9af8919f75459ca85adde47a843652 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/5b9af8919f75459ca85adde47a843652 2023-03-30 16:55:35,684 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/828e1447ec224afe8f962ebea27626ef to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/828e1447ec224afe8f962ebea27626ef 2023-03-30 16:55:35,685 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/e4fa60e7e9a74d1f9dbcbab37b52d7a6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/e4fa60e7e9a74d1f9dbcbab37b52d7a6 2023-03-30 16:55:35,686 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/989f0e0a80384ac587429e29316b6f80 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/989f0e0a80384ac587429e29316b6f80 2023-03-30 16:55:35,688 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c465cf722361474ab5b9f91775b93104 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/c465cf722361474ab5b9f91775b93104 2023-03-30 16:55:35,689 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/78a1cf7a0b024023be50eb5c57e9d792 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/78a1cf7a0b024023be50eb5c57e9d792 2023-03-30 16:55:35,690 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/051f352a0c4d40e38246e25a59ed8e62 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/051f352a0c4d40e38246e25a59ed8e62 2023-03-30 16:55:35,691 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/e9a22b4bbdfc4271a0388ba1034ab596 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/e9a22b4bbdfc4271a0388ba1034ab596 2023-03-30 16:55:35,695 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/b8c04902a5ef44cba7ac0df1a9618b18, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/59d515e3b3fe4b8dba458d99e55e1dde, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/e8bbd703c8fc408ab94f92d82cfdb81f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/782ac22386f44f83801710e76aa140e7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/3b301ea7019a4d499efda9139729978f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/c5d4d7e365b34a36963e766783856006, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/8592ff3476d54b628b12c8416a1b30b4, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/0e29618878cf466ca0656809be138503, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/baa484a67ea047caa0358af1ea77d21c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/4d0838dce9174507972ca85a717093bb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/47ed8b0684c64e62b27c0fcf186a3aea, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/1c1f1f9bcffb4721928f6d8e412e8dc8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/a65302887b5d4feab61c2ac7e4bd2897, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/a83ac8e06e644b1f9cf0e88d47d3f3e8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/1154dccb2bb84992a3421a7ee89945a7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/381daa52fdc242b59ccda3fe537cf08f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/2dd9d0025465449495f1616f4319ce82, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/6e193d869ce64607bd572ae93ff0197f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/44b902d025d9441185b6d86a04213cc7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/43d3ba2e54e340fea04060f55789d24f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/cb520ef119aa4c15a747700f66348773, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/4c6ce09a96ee49a5952fc774d22c8cb4, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/dc5f37e0301a4bd8bef24a79fbb5db30, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/bfdb5d162c654278848f514b86662524] to archive 2023-03-30 16:55:35,696 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 16:55:35,697 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/b8c04902a5ef44cba7ac0df1a9618b18 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/b8c04902a5ef44cba7ac0df1a9618b18 2023-03-30 16:55:35,698 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/59d515e3b3fe4b8dba458d99e55e1dde to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/59d515e3b3fe4b8dba458d99e55e1dde 2023-03-30 16:55:35,699 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/e8bbd703c8fc408ab94f92d82cfdb81f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/e8bbd703c8fc408ab94f92d82cfdb81f 2023-03-30 16:55:35,700 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/782ac22386f44f83801710e76aa140e7 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/782ac22386f44f83801710e76aa140e7 2023-03-30 16:55:35,701 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/3b301ea7019a4d499efda9139729978f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/3b301ea7019a4d499efda9139729978f 2023-03-30 16:55:35,702 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/c5d4d7e365b34a36963e766783856006 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/c5d4d7e365b34a36963e766783856006 2023-03-30 16:55:35,702 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/8592ff3476d54b628b12c8416a1b30b4 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/8592ff3476d54b628b12c8416a1b30b4 2023-03-30 16:55:35,703 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/0e29618878cf466ca0656809be138503 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/0e29618878cf466ca0656809be138503 2023-03-30 16:55:35,704 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/baa484a67ea047caa0358af1ea77d21c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/baa484a67ea047caa0358af1ea77d21c 2023-03-30 16:55:35,705 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/4d0838dce9174507972ca85a717093bb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/4d0838dce9174507972ca85a717093bb 2023-03-30 16:55:35,706 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/47ed8b0684c64e62b27c0fcf186a3aea to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/47ed8b0684c64e62b27c0fcf186a3aea 2023-03-30 16:55:35,706 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/1c1f1f9bcffb4721928f6d8e412e8dc8 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/1c1f1f9bcffb4721928f6d8e412e8dc8 2023-03-30 16:55:35,707 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/a65302887b5d4feab61c2ac7e4bd2897 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/a65302887b5d4feab61c2ac7e4bd2897 2023-03-30 16:55:35,708 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/a83ac8e06e644b1f9cf0e88d47d3f3e8 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/a83ac8e06e644b1f9cf0e88d47d3f3e8 2023-03-30 16:55:35,709 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/1154dccb2bb84992a3421a7ee89945a7 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/1154dccb2bb84992a3421a7ee89945a7 2023-03-30 16:55:35,710 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/381daa52fdc242b59ccda3fe537cf08f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/381daa52fdc242b59ccda3fe537cf08f 2023-03-30 16:55:35,710 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/2dd9d0025465449495f1616f4319ce82 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/2dd9d0025465449495f1616f4319ce82 2023-03-30 16:55:35,711 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/6e193d869ce64607bd572ae93ff0197f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/6e193d869ce64607bd572ae93ff0197f 2023-03-30 16:55:35,712 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/44b902d025d9441185b6d86a04213cc7 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/44b902d025d9441185b6d86a04213cc7 2023-03-30 16:55:35,713 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/43d3ba2e54e340fea04060f55789d24f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/43d3ba2e54e340fea04060f55789d24f 2023-03-30 16:55:35,714 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/cb520ef119aa4c15a747700f66348773 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/cb520ef119aa4c15a747700f66348773 2023-03-30 16:55:35,714 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/4c6ce09a96ee49a5952fc774d22c8cb4 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/4c6ce09a96ee49a5952fc774d22c8cb4 2023-03-30 16:55:35,715 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/dc5f37e0301a4bd8bef24a79fbb5db30 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/dc5f37e0301a4bd8bef24a79fbb5db30 2023-03-30 16:55:35,716 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/bfdb5d162c654278848f514b86662524 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/bfdb5d162c654278848f514b86662524 2023-03-30 16:55:35,721 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/3cf07530b0f942e687b4053caf9cb10a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/0c129f8304554bf0a18e1726c4a68640, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/6bb362391f05459babc371369e211283, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/6f50d38c154e470b9e8fdfde9db958d3, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/25aa9583403e4a35a21fddac5cdd89b3, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/a22a061eee32485d9cd4341fbb8444d5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/bb7bec612ff94cc996aa5d9777ceadc5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/f1062cb1b71645a18fdc59fd517b32df, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/63d1445b855a43ea970672a6bac76243, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/0dc79afe64b247d68bf8c28df73108f8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/3266846abdc74af7b3716d7249f7075e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/4c70867ab20143bc9c008d7ebac151d9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/8fcdbc587c68401a846dd7424bfff6b5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/3909d2f732834a86b0057b898bc75b5b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/cde10d4c9ee04482a7df5929b68fb75a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/7eb2667f0eac4fbcbc46432ca5f0c6a4, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/e92b71aa13604e8682ff93b5516345fb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/1fad2eb5eca449c2915b372cb05e7339, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/e239bce646a44666b58ac67f8376583d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/6e09bf9108434157942f20d78b3b48cb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/d9f6b6e852ae4de7af5740698e765c1e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/0ac8abf09bc84a94bdcc8a5cd9644976, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/902fe6e43d9b453a9ff87f37886caf37, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/9e27e7870e554341b929758944178515] to archive 2023-03-30 16:55:35,722 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 16:55:35,723 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/3cf07530b0f942e687b4053caf9cb10a to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/3cf07530b0f942e687b4053caf9cb10a 2023-03-30 16:55:35,723 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/0c129f8304554bf0a18e1726c4a68640 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/0c129f8304554bf0a18e1726c4a68640 2023-03-30 16:55:35,724 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/6bb362391f05459babc371369e211283 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/6bb362391f05459babc371369e211283 2023-03-30 16:55:35,725 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/6f50d38c154e470b9e8fdfde9db958d3 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/6f50d38c154e470b9e8fdfde9db958d3 2023-03-30 16:55:35,725 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/25aa9583403e4a35a21fddac5cdd89b3 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/25aa9583403e4a35a21fddac5cdd89b3 2023-03-30 16:55:35,726 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/a22a061eee32485d9cd4341fbb8444d5 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/a22a061eee32485d9cd4341fbb8444d5 2023-03-30 16:55:35,727 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/bb7bec612ff94cc996aa5d9777ceadc5 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/bb7bec612ff94cc996aa5d9777ceadc5 2023-03-30 16:55:35,728 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/f1062cb1b71645a18fdc59fd517b32df to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/f1062cb1b71645a18fdc59fd517b32df 2023-03-30 16:55:35,729 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/63d1445b855a43ea970672a6bac76243 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/63d1445b855a43ea970672a6bac76243 2023-03-30 16:55:35,730 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330b04ac571c19c438c815ee0f4f91f6804 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330b04ac571c19c438c815ee0f4f91f6804 2023-03-30 16:55:35,730 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/0dc79afe64b247d68bf8c28df73108f8 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/0dc79afe64b247d68bf8c28df73108f8 2023-03-30 16:55:35,731 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/3266846abdc74af7b3716d7249f7075e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/3266846abdc74af7b3716d7249f7075e 2023-03-30 16:55:35,732 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/4c70867ab20143bc9c008d7ebac151d9 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/4c70867ab20143bc9c008d7ebac151d9 2023-03-30 16:55:35,733 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/8fcdbc587c68401a846dd7424bfff6b5 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/8fcdbc587c68401a846dd7424bfff6b5 2023-03-30 16:55:35,734 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/3909d2f732834a86b0057b898bc75b5b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/3909d2f732834a86b0057b898bc75b5b 2023-03-30 16:55:35,735 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/cde10d4c9ee04482a7df5929b68fb75a to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/cde10d4c9ee04482a7df5929b68fb75a 2023-03-30 16:55:35,736 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/7eb2667f0eac4fbcbc46432ca5f0c6a4 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/7eb2667f0eac4fbcbc46432ca5f0c6a4 2023-03-30 16:55:35,737 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/e92b71aa13604e8682ff93b5516345fb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/e92b71aa13604e8682ff93b5516345fb 2023-03-30 16:55:35,737 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/1fad2eb5eca449c2915b372cb05e7339 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/1fad2eb5eca449c2915b372cb05e7339 2023-03-30 16:55:35,738 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/e239bce646a44666b58ac67f8376583d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/e239bce646a44666b58ac67f8376583d 2023-03-30 16:55:35,739 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/6e09bf9108434157942f20d78b3b48cb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/6e09bf9108434157942f20d78b3b48cb 2023-03-30 16:55:35,740 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/d9f6b6e852ae4de7af5740698e765c1e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/d9f6b6e852ae4de7af5740698e765c1e 2023-03-30 16:55:35,740 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/0ac8abf09bc84a94bdcc8a5cd9644976 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/0ac8abf09bc84a94bdcc8a5cd9644976 2023-03-30 16:55:35,741 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/902fe6e43d9b453a9ff87f37886caf37 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/902fe6e43d9b453a9ff87f37886caf37 2023-03-30 16:55:35,742 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/9e27e7870e554341b929758944178515 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/9e27e7870e554341b929758944178515 2023-03-30 16:55:35,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-30 16:55:36,134 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=351, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/dc65ebdf81d949689a6dfa772c1fe56d 2023-03-30 16:55:36,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-30 16:55:36,544 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/44c4d24dda7f4a84968621f96cda9ed2 2023-03-30 16:55:36,953 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/caee8a8235984eb382583e861a30c0db 2023-03-30 16:55:36,956 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/A/dc65ebdf81d949689a6dfa772c1fe56d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/dc65ebdf81d949689a6dfa772c1fe56d 2023-03-30 16:55:36,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/dc65ebdf81d949689a6dfa772c1fe56d, entries=100, sequenceid=351, filesize=18.6 K 2023-03-30 16:55:36,960 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/B/44c4d24dda7f4a84968621f96cda9ed2 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/44c4d24dda7f4a84968621f96cda9ed2 2023-03-30 16:55:36,962 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/44c4d24dda7f4a84968621f96cda9ed2, entries=100, sequenceid=351, filesize=9.5 K 2023-03-30 16:55:36,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/.tmp/C/caee8a8235984eb382583e861a30c0db as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/caee8a8235984eb382583e861a30c0db 2023-03-30 16:55:36,965 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/caee8a8235984eb382583e861a30c0db, entries=100, sequenceid=351, filesize=9.5 K 2023-03-30 16:55:36,966 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for e3fe0ff55782e6f5d2f55c861f91d3cb in 1647ms, sequenceid=351, compaction requested=false 2023-03-30 16:55:36,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/recovered.edits/354.seqid, newMaxSeqId=354, maxSeqId=4 2023-03-30 16:55:36,973 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb. 2023-03-30 16:55:36,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e3fe0ff55782e6f5d2f55c861f91d3cb: 2023-03-30 16:55:36,975 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:36,975 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=e3fe0ff55782e6f5d2f55c861f91d3cb, regionState=CLOSED 2023-03-30 16:55:36,975 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195336975"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680195336975"}]},"ts":"1680195336975"} 2023-03-30 16:55:36,977 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-03-30 16:55:36,977 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure e3fe0ff55782e6f5d2f55c861f91d3cb, server=jenkins-hbase4.apache.org,45193,1680195214285 in 1.8090 sec 2023-03-30 16:55:36,978 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-03-30 16:55:36,978 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e3fe0ff55782e6f5d2f55c861f91d3cb, UNASSIGN in 1.8130 sec 2023-03-30 16:55:36,979 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195336979"}]},"ts":"1680195336979"} 2023-03-30 16:55:36,979 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-30 16:55:36,981 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-30 16:55:36,982 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8250 sec 2023-03-30 16:55:37,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-30 16:55:37,265 INFO [Listener at localhost/38075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-03-30 16:55:37,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-30 16:55:37,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:55:37,268 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:55:37,268 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:55:37,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-30 16:55:37,271 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:37,273 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A, FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B, FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C, FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/recovered.edits] 2023-03-30 16:55:37,275 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/cd432fceef1844b3afff94011b7ee160 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/cd432fceef1844b3afff94011b7ee160 2023-03-30 16:55:37,276 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/dc65ebdf81d949689a6dfa772c1fe56d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/A/dc65ebdf81d949689a6dfa772c1fe56d 2023-03-30 16:55:37,278 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/44c4d24dda7f4a84968621f96cda9ed2 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/44c4d24dda7f4a84968621f96cda9ed2 2023-03-30 16:55:37,279 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/a64f437df89a4a1e8bea181b55b3c180 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/B/a64f437df89a4a1e8bea181b55b3c180 2023-03-30 16:55:37,281 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/77718c35daca4a7cbe28f3f8ea978b97 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/77718c35daca4a7cbe28f3f8ea978b97 2023-03-30 16:55:37,282 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/caee8a8235984eb382583e861a30c0db to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/C/caee8a8235984eb382583e861a30c0db 2023-03-30 16:55:37,284 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/recovered.edits/354.seqid to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb/recovered.edits/354.seqid 2023-03-30 16:55:37,285 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/e3fe0ff55782e6f5d2f55c861f91d3cb 2023-03-30 16:55:37,285 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 16:55:37,285 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-30 16:55:37,285 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-30 16:55:37,287 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330085da5dce2f540bcb31076226aedfa1d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330085da5dce2f540bcb31076226aedfa1d 2023-03-30 16:55:37,288 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033015f65013b2af4b8298e8075b64c0280f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033015f65013b2af4b8298e8075b64c0280f 2023-03-30 16:55:37,289 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303301a7ca0bf97514beaaa972bff2a1a0f8f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303301a7ca0bf97514beaaa972bff2a1a0f8f 2023-03-30 16:55:37,290 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303301a9d73cc31c1445a86179baf316d5555 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303301a9d73cc31c1445a86179baf316d5555 2023-03-30 16:55:37,291 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330249a4a7f90114e12b88f287cf7debef6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330249a4a7f90114e12b88f287cf7debef6 2023-03-30 16:55:37,292 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303302f8bf8d119974ae3a4657ce996ff5bf3 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303302f8bf8d119974ae3a4657ce996ff5bf3 2023-03-30 16:55:37,292 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303104cbe9b452463bb8fab71d84a4c7d6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303104cbe9b452463bb8fab71d84a4c7d6 2023-03-30 16:55:37,293 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303e044154fbcd46949396628847441017 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303303e044154fbcd46949396628847441017 2023-03-30 16:55:37,294 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303304dc7c8679fe740799376f8deadfcbe4c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303304dc7c8679fe740799376f8deadfcbe4c 2023-03-30 16:55:37,295 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330708d75b02adf4bd589a394ff5433ae7b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330708d75b02adf4bd589a394ff5433ae7b 2023-03-30 16:55:37,296 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033070de98dd096746aa9f0fc89032f67dc9 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033070de98dd096746aa9f0fc89032f67dc9 2023-03-30 16:55:37,296 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303307144daa2e7594e7ab2692db497c3675e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303307144daa2e7594e7ab2692db497c3675e 2023-03-30 16:55:37,297 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303308bfd9c67802041388ed9df6d780bc9df to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303308bfd9c67802041388ed9df6d780bc9df 2023-03-30 16:55:37,298 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033090352b4a37fa4f0183db06c0872666fd to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033090352b4a37fa4f0183db06c0872666fd 2023-03-30 16:55:37,299 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330b04ac571c19c438c815ee0f4f91f6804 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330b04ac571c19c438c815ee0f4f91f6804 2023-03-30 16:55:37,300 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330b65585e825d34ce488e7eebd4b26cb64 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330b65585e825d34ce488e7eebd4b26cb64 2023-03-30 16:55:37,301 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330c003e20b9e1042989d60cc3f80773abc to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330c003e20b9e1042989d60cc3f80773abc 2023-03-30 16:55:37,302 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330cfa61fe2cc784940a81e50d0beb49243 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330cfa61fe2cc784940a81e50d0beb49243 2023-03-30 16:55:37,302 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d7e79a11485b44678fb2ffe4c7328927 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d7e79a11485b44678fb2ffe4c7328927 2023-03-30 16:55:37,303 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330db7223a37f91481eae2be0775a2b800e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330db7223a37f91481eae2be0775a2b800e 2023-03-30 16:55:37,304 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330dc5d013dd806417c8eaea72472398017 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330dc5d013dd806417c8eaea72472398017 2023-03-30 16:55:37,304 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-30 16:55:37,306 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:55:37,307 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-30 16:55:37,309 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-30 16:55:37,309 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:55:37,309 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-30 16:55:37,309 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680195337309"}]},"ts":"9223372036854775807"} 2023-03-30 16:55:37,311 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-30 16:55:37,311 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => e3fe0ff55782e6f5d2f55c861f91d3cb, NAME => 'TestAcidGuarantees,,1680195302154.e3fe0ff55782e6f5d2f55c861f91d3cb.', STARTKEY => '', ENDKEY => ''}] 2023-03-30 16:55:37,311 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-30 16:55:37,311 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680195337311"}]},"ts":"9223372036854775807"} 2023-03-30 16:55:37,312 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-30 16:55:37,317 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:55:37,318 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 52 msec 2023-03-30 16:55:37,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-30 16:55:37,369 INFO [Listener at localhost/38075] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-03-30 16:55:37,380 INFO [Listener at localhost/38075] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=229 (was 228) - Thread LEAK? -, OpenFileDescriptor=517 (was 535), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=365 (was 416), ProcessCount=169 (was 169), AvailableMemoryMB=4351 (was 4513) 2023-03-30 16:55:37,391 INFO [Listener at localhost/38075] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229, OpenFileDescriptor=517, MaxFileDescriptor=60000, SystemLoadAverage=365, ProcessCount=169, AvailableMemoryMB=4351 2023-03-30 16:55:37,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-30 16:55:37,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-30 16:55:37,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-30 16:55:37,395 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-30 16:55:37,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-03-30 16:55:37,396 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-30 16:55:37,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-30 16:55:37,397 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:37,397 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79 empty. 2023-03-30 16:55:37,398 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:37,398 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 16:55:37,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-30 16:55:37,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-30 16:55:37,804 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-30 16:55:37,806 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4038776cda6db2eaa29896eab9262a79, NAME => 'TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp 2023-03-30 16:55:37,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-30 16:55:38,213 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:55:38,213 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 4038776cda6db2eaa29896eab9262a79, disabling compactions & flushes 2023-03-30 16:55:38,213 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:38,213 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:38,213 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. after waiting 0 ms 2023-03-30 16:55:38,213 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:38,213 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:38,213 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:38,215 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-30 16:55:38,216 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195338216"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680195338216"}]},"ts":"1680195338216"} 2023-03-30 16:55:38,217 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-30 16:55:38,217 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-30 16:55:38,217 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195338217"}]},"ts":"1680195338217"} 2023-03-30 16:55:38,218 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-30 16:55:38,220 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4038776cda6db2eaa29896eab9262a79, ASSIGN}] 2023-03-30 16:55:38,222 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4038776cda6db2eaa29896eab9262a79, ASSIGN 2023-03-30 16:55:38,222 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=4038776cda6db2eaa29896eab9262a79, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,45193,1680195214285; forceNewPlan=false, retain=false 2023-03-30 16:55:38,374 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=4038776cda6db2eaa29896eab9262a79, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:38,374 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195338373"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680195338373"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680195338373"}]},"ts":"1680195338373"} 2023-03-30 16:55:38,375 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285}] 2023-03-30 16:55:38,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-30 16:55:38,529 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:38,529 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4038776cda6db2eaa29896eab9262a79, NAME => 'TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.', STARTKEY => '', ENDKEY => ''} 2023-03-30 16:55:38,529 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:38,529 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:55:38,529 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:38,529 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:38,530 INFO [StoreOpener-4038776cda6db2eaa29896eab9262a79-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:38,531 DEBUG [StoreOpener-4038776cda6db2eaa29896eab9262a79-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A 2023-03-30 16:55:38,531 DEBUG [StoreOpener-4038776cda6db2eaa29896eab9262a79-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A 2023-03-30 16:55:38,532 INFO [StoreOpener-4038776cda6db2eaa29896eab9262a79-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:55:38,532 INFO [StoreOpener-4038776cda6db2eaa29896eab9262a79-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4038776cda6db2eaa29896eab9262a79 columnFamilyName A 2023-03-30 16:55:38,532 INFO [StoreOpener-4038776cda6db2eaa29896eab9262a79-1] regionserver.HStore(310): Store=4038776cda6db2eaa29896eab9262a79/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:55:38,532 INFO [StoreOpener-4038776cda6db2eaa29896eab9262a79-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:38,533 DEBUG [StoreOpener-4038776cda6db2eaa29896eab9262a79-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B 2023-03-30 16:55:38,533 DEBUG [StoreOpener-4038776cda6db2eaa29896eab9262a79-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B 2023-03-30 16:55:38,533 INFO [StoreOpener-4038776cda6db2eaa29896eab9262a79-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:55:38,534 INFO [StoreOpener-4038776cda6db2eaa29896eab9262a79-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4038776cda6db2eaa29896eab9262a79 columnFamilyName B 2023-03-30 16:55:38,534 INFO [StoreOpener-4038776cda6db2eaa29896eab9262a79-1] regionserver.HStore(310): Store=4038776cda6db2eaa29896eab9262a79/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:55:38,534 INFO [StoreOpener-4038776cda6db2eaa29896eab9262a79-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:38,535 DEBUG [StoreOpener-4038776cda6db2eaa29896eab9262a79-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C 2023-03-30 16:55:38,535 DEBUG [StoreOpener-4038776cda6db2eaa29896eab9262a79-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C 2023-03-30 16:55:38,535 INFO [StoreOpener-4038776cda6db2eaa29896eab9262a79-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:55:38,535 INFO [StoreOpener-4038776cda6db2eaa29896eab9262a79-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4038776cda6db2eaa29896eab9262a79 columnFamilyName C 2023-03-30 16:55:38,536 INFO [StoreOpener-4038776cda6db2eaa29896eab9262a79-1] regionserver.HStore(310): Store=4038776cda6db2eaa29896eab9262a79/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:55:38,536 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:38,536 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:38,536 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:38,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 16:55:38,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:38,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-30 16:55:38,540 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4038776cda6db2eaa29896eab9262a79; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73024941, jitterRate=0.08815641701221466}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 16:55:38,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:38,541 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., pid=49, masterSystemTime=1680195338526 2023-03-30 16:55:38,542 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:38,542 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:38,542 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=4038776cda6db2eaa29896eab9262a79, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:38,542 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195338542"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680195338542"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680195338542"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680195338542"}]},"ts":"1680195338542"} 2023-03-30 16:55:38,544 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-03-30 16:55:38,544 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 in 168 msec 2023-03-30 16:55:38,545 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-03-30 16:55:38,545 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4038776cda6db2eaa29896eab9262a79, ASSIGN in 324 msec 2023-03-30 16:55:38,546 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-30 16:55:38,546 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195338546"}]},"ts":"1680195338546"} 2023-03-30 16:55:38,546 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-30 16:55:38,549 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-30 16:55:38,550 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1550 sec 2023-03-30 16:55:39,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-30 16:55:39,501 INFO [Listener at localhost/38075] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-03-30 16:55:39,501 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x77d7b49f to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:39,506 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b0c1677, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:39,507 DEBUG [Listener at localhost/38075] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:39,509 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57180, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:39,510 DEBUG [Listener at localhost/38075] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-30 16:55:39,511 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39904, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-30 16:55:39,512 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x2c8fbc35 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:39,516 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66369677, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:39,516 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x3fb41c50 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:39,520 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e0b6187, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:39,520 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x357cb237 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:39,524 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ab56e54, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:39,524 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x4539ed0c to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:39,527 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15840296, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:39,527 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x32cd672d to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:39,531 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3816bdf5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:39,531 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x50150525 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:39,535 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5de1e3c7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:39,536 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x028a3a77 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:39,540 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a4e3b2e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:39,540 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x4cefb012 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:39,544 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f65627a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:39,544 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x24a917df to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:39,547 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15f74c78, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:39,548 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x78fa3dee to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:55:39,552 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1401cac1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:55:39,555 DEBUG [hconnection-0x40c18922-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:39,557 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57194, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:39,558 DEBUG [hconnection-0x36b53294-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:39,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:39,558 DEBUG [hconnection-0x75d8e8b5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:39,559 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57208, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:39,559 DEBUG [hconnection-0x6e655efd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:39,560 DEBUG [hconnection-0x71aa7480-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:39,564 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57214, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:39,564 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57230, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:39,564 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57238, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:39,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:39,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:39,565 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:39,565 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:39,565 DEBUG [hconnection-0x2ac1f732-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:39,565 DEBUG [hconnection-0x21e25c6a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:39,565 DEBUG [hconnection-0x1112132a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:39,565 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:39,565 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:39,565 DEBUG [hconnection-0x2cdd4b0f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:39,565 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:39,565 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:39,566 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57264, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:39,567 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:39,568 DEBUG [hconnection-0xe8f310d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:55:39,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:39,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:39,569 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57254, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:39,568 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:39,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:39,571 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57252, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:39,571 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57278, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:39,577 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:39,577 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:39,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:39,580 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57280, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:55:39,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:39,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:39,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:39,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:39,583 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:39,583 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:39,583 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:39,584 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:39,584 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:39,584 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:39,584 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:39,588 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-30 16:55:39,588 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:39,589 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:39,589 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:39,589 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:39,589 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:39,589 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:39,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:39,590 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:39,597 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/b640b22151534e0186ba76da6ca56a63 2023-03-30 16:55:39,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1680195339590, max timestamp=1680195339608Num uniques 859; 2023-03-30 16:55:39,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680195339590, max timestamp=1680195339608Num uniques 859; 2023-03-30 16:55:39,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-30 16:55:39,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-30 16:55:39,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680195339590, max timestamp=1680195339608Num uniques 859; 2023-03-30 16:55:39,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-30 16:55:39,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:39,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195399623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:39,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:39,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195399627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:39,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:39,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195399629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:39,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:39,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195399629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:39,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:39,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195399630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:39,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:39,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195399731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:39,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:39,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195399731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:39,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:39,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195399736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:39,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:39,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195399736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:39,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:39,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195399738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:39,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:39,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195399937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:39,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:39,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195399937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:39,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:39,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195399940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:39,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:39,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195399941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:39,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:39,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195399943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,006 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/fe0353d26bab450b9c69e02f6ef558c2 2023-03-30 16:55:40,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:40,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195400242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:40,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195400243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:40,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195400246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:40,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195400247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:40,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195400248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,416 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/ad80ee225cae4268add354f6817a6095 2023-03-30 16:55:40,420 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/b640b22151534e0186ba76da6ca56a63 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/b640b22151534e0186ba76da6ca56a63 2023-03-30 16:55:40,423 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/b640b22151534e0186ba76da6ca56a63, entries=150, sequenceid=10, filesize=11.6 K 2023-03-30 16:55:40,424 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/fe0353d26bab450b9c69e02f6ef558c2 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/fe0353d26bab450b9c69e02f6ef558c2 2023-03-30 16:55:40,427 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/fe0353d26bab450b9c69e02f6ef558c2, entries=150, sequenceid=10, filesize=11.6 K 2023-03-30 16:55:40,428 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/ad80ee225cae4268add354f6817a6095 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ad80ee225cae4268add354f6817a6095 2023-03-30 16:55:40,430 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ad80ee225cae4268add354f6817a6095, entries=150, sequenceid=10, filesize=11.6 K 2023-03-30 16:55:40,431 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 4038776cda6db2eaa29896eab9262a79 in 843ms, sequenceid=10, compaction requested=false 2023-03-30 16:55:40,431 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:40,431 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:40,431 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:40,431 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:40,431 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,431 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:40,431 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:40,433 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,433 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:40,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:40,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:40,434 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:40,434 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:40,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:40,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:40,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:40,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:40,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:40,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@602a26f1[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:40,435 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:40,435 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:40,436 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:40,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:40,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:40,436 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:40,436 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:40,436 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:40,437 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,437 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:40,438 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,438 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:40,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:40,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:40,438 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:40,439 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:40,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:40,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:40,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:40,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:40,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:40,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:40,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:40,441 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:40,441 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:40,441 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:40,441 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:40,441 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:40,444 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:40,444 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:40,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:40,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:40,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:40,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:40,444 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:40,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:40,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:40,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:40,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:40,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:40,445 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:40,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:40,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:40,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,446 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:40,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:40,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,448 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:40,448 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,448 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:40,448 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:40,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:40,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:40,448 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:40,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:40,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:40,448 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:40,449 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,449 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:40,449 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:40,449 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:40,449 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:40,449 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:40,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:40,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:40,549 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:40,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:40,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:40,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:40,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:40,552 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:40,552 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:40,552 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:40,552 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:40,552 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:40,552 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:40,554 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,554 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:40,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:40,554 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,554 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:40,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:40,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:40,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:40,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:40,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:40,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:40,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:40,555 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:40,555 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:40,555 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:40,555 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:40,555 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:40,555 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:40,555 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:40,556 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-03-30 16:55:40,556 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:40,556 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:40,556 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:40,556 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:40,556 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:40,556 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:40,572 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/75574a42e1b94f4db62b4b474a8488bf 2023-03-30 16:55:40,583 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/43f4293fb14140548bf6bfaa0cdd2e1a 2023-03-30 16:55:40,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:40,754 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:40,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:40,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:40,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195400756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195400757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:40,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195400762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:40,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195400762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:40,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195400762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:40,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195400863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:40,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195400863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:40,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195400868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:40,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195400868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:40,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195400869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:40,998 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/e179218f8e854620be03f6ecb190db11 2023-03-30 16:55:41,003 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/75574a42e1b94f4db62b4b474a8488bf as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/75574a42e1b94f4db62b4b474a8488bf 2023-03-30 16:55:41,006 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/75574a42e1b94f4db62b4b474a8488bf, entries=150, sequenceid=46, filesize=11.6 K 2023-03-30 16:55:41,007 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/43f4293fb14140548bf6bfaa0cdd2e1a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/43f4293fb14140548bf6bfaa0cdd2e1a 2023-03-30 16:55:41,011 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/43f4293fb14140548bf6bfaa0cdd2e1a, entries=150, sequenceid=46, filesize=11.6 K 2023-03-30 16:55:41,011 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/e179218f8e854620be03f6ecb190db11 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/e179218f8e854620be03f6ecb190db11 2023-03-30 16:55:41,015 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/e179218f8e854620be03f6ecb190db11, entries=150, sequenceid=46, filesize=11.6 K 2023-03-30 16:55:41,015 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=46.96 KB/48090 for 4038776cda6db2eaa29896eab9262a79 in 459ms, sequenceid=46, compaction requested=false 2023-03-30 16:55:41,015 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:41,015 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:41,016 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:41,019 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:41,019 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,019 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:41,019 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:41,020 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,020 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,021 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,021 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:41,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:41,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bbe7773[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:41,023 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:41,023 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,026 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,026 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:41,026 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,026 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:41,026 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:41,026 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:41,027 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,027 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:41,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,027 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:41,027 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:41,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:41,030 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:41,030 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:41,030 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:41,030 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:41,030 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:41,031 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,031 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,032 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:41,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:41,032 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,033 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,035 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,035 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,035 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,035 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,035 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:41,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:41,035 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,035 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,035 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:41,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:41,035 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:41,035 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:41,036 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:41,036 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:41,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:41,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:41,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:41,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:41,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:41,069 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 16:55:41,070 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:41,070 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:41,070 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:41,070 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:41,070 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:41,071 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:41,084 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/ec6c20b1db7e4451890454e409970902 2023-03-30 16:55:41,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:41,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195401118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:41,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195401119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:41,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195401119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:41,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195401119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:41,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195401125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,136 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:41,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:41,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:41,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:41,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:41,139 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:41,139 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:41,139 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:41,139 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:41,139 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,139 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,141 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:41,141 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:41,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:41,142 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,142 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:41,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:41,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:41,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:41,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:41,143 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:41,143 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:41,143 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:41,143 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:41,143 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:41,143 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:41,144 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:41,144 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:41,144 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:41,144 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:41,144 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:41,144 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,144 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:41,144 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:41,145 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,145 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:41,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:41,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4480551a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:41,147 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:41,147 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,148 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,148 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:41,149 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,149 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:41,149 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:41,149 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:41,153 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,153 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:41,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,153 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:41,153 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:41,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:41,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:41,156 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:41,156 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:41,156 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:41,156 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:41,156 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:41,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:41,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:41,158 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:41,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,160 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,162 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,162 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,162 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:41,163 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:41,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:41,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:41,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:41,163 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:41,163 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:41,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:41,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:41,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:41,163 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,163 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:41,163 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,163 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:41,163 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,163 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:41,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195401226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:41,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195401226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:41,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195401228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:41,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195401228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:41,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195401228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,263 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:41,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:41,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:41,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:41,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:41,267 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:41,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:41,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:41,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:41,268 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,268 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,272 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:41,272 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:41,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:41,272 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,272 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:41,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:41,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:41,274 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:41,274 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:41,274 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:41,274 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:41,274 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:41,274 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:41,274 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:41,274 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:41,274 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:41,274 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:41,274 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:41,274 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:41,274 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:41,274 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,274 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:41,274 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:41,276 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,276 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,276 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,276 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:41,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:41,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef4c6a9[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:41,277 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:41,277 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,279 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,279 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:41,279 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,279 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:41,279 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:41,279 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:41,281 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,281 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:41,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,281 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:41,281 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:41,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:41,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:41,283 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:41,283 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:41,283 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:41,283 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:41,283 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:41,285 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,285 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,285 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:41,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:41,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:41,286 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,287 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,289 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:41,289 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:41,289 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:41,289 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:41,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:41,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:41,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:41,289 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,289 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,289 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,289 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,289 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:41,289 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:41,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:41,289 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,290 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:41,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:41,389 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:41,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:41,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:41,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:41,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:41,393 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:41,393 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:41,393 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:41,393 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:41,393 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,393 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,395 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,395 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:41,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:41,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:41,395 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,395 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:41,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:41,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:41,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:41,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:41,396 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:41,396 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:41,396 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:41,397 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:41,397 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:41,397 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:41,397 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:41,397 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:41,397 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:41,397 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:41,397 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:41,397 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,397 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:41,397 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:41,399 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,399 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,399 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,399 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:41,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:41,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f586a28[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:41,400 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:41,400 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,403 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,403 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:41,403 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:41,403 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:41,403 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,403 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:41,404 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:41,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:41,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:41,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:41,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:41,407 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:41,407 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:41,407 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:41,407 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:41,407 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:41,410 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,410 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,410 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:41,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:41,410 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:41,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,412 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,414 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,414 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:41,414 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:41,414 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:41,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:41,414 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,414 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:41,414 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:41,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:41,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:41,414 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,414 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,414 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,414 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:41,414 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:41,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:41,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:41,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:41,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195401430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:41,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:41,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195401431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195401433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:41,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195401434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:41,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195401434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,495 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/6988fcd6629d47be9487356258e156a6 2023-03-30 16:55:41,515 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:41,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:41,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:41,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:41,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:41,518 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:41,518 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:41,518 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:41,518 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:41,518 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,518 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,520 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:41,520 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:41,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:41,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,520 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,520 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:41,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:41,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:41,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:41,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:41,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:41,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:41,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:41,522 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:41,522 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:41,522 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:41,522 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:41,522 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:41,522 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:41,522 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:41,522 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:41,522 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,522 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:41,522 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:41,523 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,524 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,524 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:41,524 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:41,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67f5895f[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:41,526 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:41,526 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,528 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,528 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:41,528 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:41,528 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,528 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:41,528 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:41,530 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,530 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:41,530 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:41,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,530 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:41,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:41,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:41,532 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:41,532 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:41,532 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:41,532 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:41,532 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:41,537 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,537 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,537 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:41,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,538 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,539 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,541 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,541 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,541 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:41,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:41,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:41,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:41,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:41,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:41,541 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:41,541 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:41,541 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,541 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:41,542 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,542 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:41,542 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:41,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:41,542 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,542 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,642 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:41,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:41,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:41,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:41,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:41,649 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:41,649 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:41,649 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:41,649 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:41,649 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,649 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,651 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:41,651 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:41,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:41,651 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,651 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:41,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:41,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:41,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:41,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:41,652 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:41,652 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:41,652 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:41,652 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:41,652 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:41,652 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:41,652 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:41,652 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:41,652 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:41,652 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:41,652 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:41,652 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,653 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:41,653 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:41,655 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,655 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,655 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,655 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:41,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:41,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@211b383e[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:41,657 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:41,657 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,658 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,658 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:41,658 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:41,658 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:41,658 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,658 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:41,659 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,659 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:41,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,659 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:41,659 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:41,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:41,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:41,662 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:41,662 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:41,662 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:41,662 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:41,662 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:41,664 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,664 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,664 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:41,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:41,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:41,664 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,665 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,668 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:41,668 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:41,668 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,668 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,668 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:41,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:41,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:41,668 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:41,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:41,668 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:41,669 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:41,669 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:41,669 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:41,669 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:41,669 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:41,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195401741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:41,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195401741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:41,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195401741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:41,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195401741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:41,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195401741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,769 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:41,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:41,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:41,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:41,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:41,772 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:41,772 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:41,772 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:41,772 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:41,772 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,772 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,774 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:41,774 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:41,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:41,774 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,774 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:41,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:41,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:41,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:41,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:41,775 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:41,775 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:41,775 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:41,775 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:41,775 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:41,775 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:41,775 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:41,776 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:41,776 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:41,776 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:41,776 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:41,776 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,776 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:41,776 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:41,777 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,777 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,778 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,778 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:41,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:41,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7982733a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:41,779 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:41,779 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,780 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,780 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:41,780 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:41,780 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:41,780 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,780 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:41,782 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,782 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:41,782 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:41,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,782 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:41,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:41,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:41,784 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:41,784 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:41,784 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:41,784 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:41,784 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:41,785 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,785 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,786 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:41,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:41,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:41,786 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,787 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,789 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,789 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,789 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,792 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:41,792 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:41,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:41,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:41,792 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:41,792 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:41,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:41,792 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,792 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:41,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:41,792 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,804 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:41,804 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:41,904 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:41,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:41,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:41,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:41,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:41,907 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:41,907 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/5c493d60f5ee4599b13f8d29c20abae0 2023-03-30 16:55:41,907 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:41,907 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:41,907 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:41,908 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,908 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,909 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:41,909 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:41,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,910 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,910 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:41,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:41,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:41,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:41,911 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:41,911 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:41,911 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:41,911 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:41,911 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:41,911 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:41,911 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:41,911 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:41,911 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:41,911 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:41,911 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/ec6c20b1db7e4451890454e409970902 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/ec6c20b1db7e4451890454e409970902 2023-03-30 16:55:41,911 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:41,912 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,912 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:41,912 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:41,914 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,914 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,914 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,914 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:41,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,915 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/ec6c20b1db7e4451890454e409970902, entries=250, sequenceid=58, filesize=16.2 K 2023-03-30 16:55:41,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:41,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75075f08[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:41,916 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/6988fcd6629d47be9487356258e156a6 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/6988fcd6629d47be9487356258e156a6 2023-03-30 16:55:41,916 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:41,916 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,920 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,920 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:41,920 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:41,920 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:41,920 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,920 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:41,922 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,922 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:41,922 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:41,922 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,924 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/6988fcd6629d47be9487356258e156a6, entries=150, sequenceid=58, filesize=11.6 K 2023-03-30 16:55:41,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:41,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:41,924 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:41,925 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:41,925 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:41,925 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:41,925 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:41,925 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/5c493d60f5ee4599b13f8d29c20abae0 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/5c493d60f5ee4599b13f8d29c20abae0 2023-03-30 16:55:41,928 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,928 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:41,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:41,928 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:41,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:41,928 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:41,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:41,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:41,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,930 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:41,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:41,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,931 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/5c493d60f5ee4599b13f8d29c20abae0, entries=150, sequenceid=58, filesize=11.6 K 2023-03-30 16:55:41,931 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4038776cda6db2eaa29896eab9262a79 in 862ms, sequenceid=58, compaction requested=true 2023-03-30 16:55:41,931 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:41,932 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:41,932 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-30 16:55:41,932 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-30 16:55:41,932 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:55:41,932 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:55:41,933 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40338 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:55:41,933 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/A is initiating minor compaction (all files) 2023-03-30 16:55:41,933 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/A in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:41,933 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/b640b22151534e0186ba76da6ca56a63, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/75574a42e1b94f4db62b4b474a8488bf, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/ec6c20b1db7e4451890454e409970902] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=39.4 K 2023-03-30 16:55:41,933 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:55:41,933 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/C is initiating minor compaction (all files) 2023-03-30 16:55:41,933 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/C in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:41,933 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ad80ee225cae4268add354f6817a6095, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/e179218f8e854620be03f6ecb190db11, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/5c493d60f5ee4599b13f8d29c20abae0] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=34.8 K 2023-03-30 16:55:41,934 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting ad80ee225cae4268add354f6817a6095, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1680195339582 2023-03-30 16:55:41,934 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting b640b22151534e0186ba76da6ca56a63, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1680195339582 2023-03-30 16:55:41,934 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting e179218f8e854620be03f6ecb190db11, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680195339627 2023-03-30 16:55:41,934 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 75574a42e1b94f4db62b4b474a8488bf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680195339627 2023-03-30 16:55:41,934 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 5c493d60f5ee4599b13f8d29c20abae0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680195340756 2023-03-30 16:55:41,934 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting ec6c20b1db7e4451890454e409970902, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680195340756 2023-03-30 16:55:41,937 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,937 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,937 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:41,937 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:41,937 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,937 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:41,937 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:41,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:41,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:41,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:41,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:41,937 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:41,937 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:41,937 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:41,937 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:41,937 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:41,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:41,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:41,942 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#C#compaction#359 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:41,942 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#A#compaction#360 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:41,963 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/a35125cf1c424f879d9640edaab88071 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/a35125cf1c424f879d9640edaab88071 2023-03-30 16:55:41,972 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/ad3417c1cb274c6699ea42f4a1608257 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ad3417c1cb274c6699ea42f4a1608257 2023-03-30 16:55:41,985 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/C of 4038776cda6db2eaa29896eab9262a79 into ad3417c1cb274c6699ea42f4a1608257(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:41,985 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:41,986 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/C, priority=13, startTime=1680195341932; duration=0sec 2023-03-30 16:55:41,986 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:41,986 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:55:41,988 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:55:41,988 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/B is initiating minor compaction (all files) 2023-03-30 16:55:41,988 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/B in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:41,988 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/fe0353d26bab450b9c69e02f6ef558c2, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/43f4293fb14140548bf6bfaa0cdd2e1a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/6988fcd6629d47be9487356258e156a6] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=34.8 K 2023-03-30 16:55:41,988 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/A of 4038776cda6db2eaa29896eab9262a79 into a35125cf1c424f879d9640edaab88071(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:41,988 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:41,988 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/A, priority=13, startTime=1680195341932; duration=0sec 2023-03-30 16:55:41,988 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:41,989 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting fe0353d26bab450b9c69e02f6ef558c2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1680195339582 2023-03-30 16:55:41,989 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 43f4293fb14140548bf6bfaa0cdd2e1a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680195339627 2023-03-30 16:55:41,989 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 6988fcd6629d47be9487356258e156a6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680195340756 2023-03-30 16:55:42,004 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#B#compaction#361 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:42,038 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:42,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:42,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:42,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:42,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:42,041 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:42,041 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:42,041 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:42,041 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:42,041 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:42,041 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:42,043 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,043 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:42,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:42,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:42,043 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,043 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:42,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:42,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:42,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:42,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:42,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:42,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:42,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:42,044 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:42,044 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:42,044 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:42,044 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:42,044 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:42,044 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:42,045 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:42,045 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-30 16:55:42,045 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:42,045 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:42,045 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:42,045 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:42,045 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:42,045 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:42,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:42,247 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:42,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:42,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195402253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:42,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195402253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:42,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195402254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:42,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195402257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:42,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195402257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:42,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195402359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:42,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195402359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:42,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195402360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:42,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195402360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:42,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195402360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,422 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/5685f49a3f82405bad04127e8d9985ac as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/5685f49a3f82405bad04127e8d9985ac 2023-03-30 16:55:42,426 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/B of 4038776cda6db2eaa29896eab9262a79 into 5685f49a3f82405bad04127e8d9985ac(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:42,426 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:42,426 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/B, priority=13, startTime=1680195341932; duration=0sec 2023-03-30 16:55:42,426 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:42,462 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/bed124d544914fdb9ee870765f1048dc 2023-03-30 16:55:42,512 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-30 16:55:42,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:42,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195402563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:42,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195402563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:42,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195402565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:42,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195402566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:42,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195402566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:42,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195402868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:42,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195402869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:42,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195402869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:42,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195402870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:42,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195402870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,874 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/6ba3d430cfe3496a9edddc1600c24892 2023-03-30 16:55:42,882 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/59500937e31646708ad766f6fd36523c 2023-03-30 16:55:42,886 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/bed124d544914fdb9ee870765f1048dc as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/bed124d544914fdb9ee870765f1048dc 2023-03-30 16:55:42,889 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/bed124d544914fdb9ee870765f1048dc, entries=150, sequenceid=80, filesize=11.6 K 2023-03-30 16:55:42,890 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/6ba3d430cfe3496a9edddc1600c24892 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/6ba3d430cfe3496a9edddc1600c24892 2023-03-30 16:55:42,896 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/6ba3d430cfe3496a9edddc1600c24892, entries=150, sequenceid=80, filesize=11.6 K 2023-03-30 16:55:42,897 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/59500937e31646708ad766f6fd36523c as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/59500937e31646708ad766f6fd36523c 2023-03-30 16:55:42,900 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/59500937e31646708ad766f6fd36523c, entries=150, sequenceid=80, filesize=11.6 K 2023-03-30 16:55:42,900 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 4038776cda6db2eaa29896eab9262a79 in 855ms, sequenceid=80, compaction requested=false 2023-03-30 16:55:42,900 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:42,901 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:42,902 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:42,903 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:42,903 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,903 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:42,903 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:42,904 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,904 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:42,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:42,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:42,905 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:42,905 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:42,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:42,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:42,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:42,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:42,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:42,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ef38f86[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:42,906 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:42,906 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:42,907 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:42,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:42,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:42,907 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:42,908 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:42,908 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:42,908 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,908 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:42,910 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,910 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:42,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:42,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:42,910 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:42,911 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:42,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:42,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:42,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:42,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:42,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:42,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:42,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:42,913 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:42,914 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:42,914 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:42,914 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:42,914 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:42,915 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:42,915 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:42,915 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:42,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:42,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:42,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:42,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:42,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:42,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:42,916 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:42,917 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:42,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,919 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:42,919 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:42,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:42,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:42,919 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:42,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:42,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:42,919 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:42,919 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:42,919 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:42,919 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:42,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:42,919 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:42,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:42,919 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:42,919 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:42,919 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:42,919 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,019 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:43,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:43,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:43,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:43,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:43,025 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:43,025 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:43,025 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:43,025 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:43,026 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,026 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:43,027 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,027 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:43,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:43,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:43,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:43,028 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,028 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:43,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:43,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:43,028 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:43,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:43,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:43,029 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:43,029 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:43,029 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:43,029 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:43,029 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:43,029 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:43,029 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:43,029 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-30 16:55:43,029 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:43,029 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:43,030 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:43,030 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:43,030 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:43,030 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:43,047 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/1d529e778f9a40f08f49bce3609b0710 2023-03-30 16:55:43,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:43,376 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:43,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:43,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195403395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:43,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195403396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:43,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195403397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:43,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195403397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:43,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195403402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,466 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/74e38e865078432ca2578c1454cba9ad 2023-03-30 16:55:43,475 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/7b40eaa9ae374f4b9612f7d9067a4c75 2023-03-30 16:55:43,479 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/1d529e778f9a40f08f49bce3609b0710 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/1d529e778f9a40f08f49bce3609b0710 2023-03-30 16:55:43,482 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/1d529e778f9a40f08f49bce3609b0710, entries=150, sequenceid=93, filesize=11.6 K 2023-03-30 16:55:43,483 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/74e38e865078432ca2578c1454cba9ad as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/74e38e865078432ca2578c1454cba9ad 2023-03-30 16:55:43,497 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/74e38e865078432ca2578c1454cba9ad, entries=150, sequenceid=93, filesize=11.6 K 2023-03-30 16:55:43,498 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/7b40eaa9ae374f4b9612f7d9067a4c75 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/7b40eaa9ae374f4b9612f7d9067a4c75 2023-03-30 16:55:43,501 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/7b40eaa9ae374f4b9612f7d9067a4c75, entries=150, sequenceid=93, filesize=11.6 K 2023-03-30 16:55:43,502 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4038776cda6db2eaa29896eab9262a79 in 473ms, sequenceid=93, compaction requested=true 2023-03-30 16:55:43,502 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:43,502 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:43,502 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:43,502 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:43,502 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,502 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:43,502 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:43,505 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,505 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:43,505 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,505 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:43,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195343395, max timestamp=1680195343506Num uniques 859; 2023-03-30 16:55:43,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:43,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195343395, max timestamp=1680195343506Num uniques 859; 2023-03-30 16:55:43,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195343395, max timestamp=1680195343506Num uniques 859; 2023-03-30 16:55:43,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:55:43,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:55:43,508 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=89.45 KB heapSize=262.11 KB 2023-03-30 16:55:43,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:55:43,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:43,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46d18aae[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:43,508 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:43,508 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,509 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:43,509 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:43,509 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:43,509 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:43,509 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:43,509 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:43,513 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,513 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,513 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:43,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,514 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:43,514 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:43,514 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:43,516 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:43,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:43,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:43,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:43,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:43,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:43,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:43,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:43,522 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:43,522 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:43,522 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:43,522 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:43,522 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:43,524 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,524 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:43,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,524 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:43,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:43,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,525 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,529 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:43,529 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:43,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:43,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:43,529 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:43,529 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:43,529 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:43,529 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:43,529 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:43,529 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,529 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,529 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:43,529 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,529 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:43,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:43,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:43,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195343532, max timestamp=1680195343544Num uniques 859; 2023-03-30 16:55:43,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:55:43,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195343532, max timestamp=1680195343544Num uniques 859; 2023-03-30 16:55:43,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195343532, max timestamp=1680195343544Num uniques 859; 2023-03-30 16:55:43,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:55:43,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:55:43,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:43,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195403574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680195343563, max timestamp=1680195343582Num uniques 1018; 2023-03-30 16:55:43,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-30 16:55:43,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680195343563, max timestamp=1680195343582Num uniques 1018; 2023-03-30 16:55:43,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680195343563, max timestamp=1680195343582Num uniques 1018; 2023-03-30 16:55:43,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-30 16:55:43,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-30 16:55:43,619 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:43,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195403610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:43,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195403612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:43,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195403618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:43,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195403619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,629 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:43,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:43,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:43,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:43,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:43,636 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:43,636 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:43,636 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:43,636 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:43,639 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,639 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:43,640 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:43,640 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:43,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:43,641 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,641 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:43,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:43,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:43,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:43,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:43,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:43,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:43,642 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:43,643 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:43,643 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:43,648 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:43,648 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:43,648 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:43,648 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:43,648 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:43,648 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:43,648 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:43,648 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:43,648 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,648 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:43,648 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:43,650 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,650 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:43,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:43,650 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,650 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:43,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:43,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:43,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:43,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:43,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ca03aed[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:43,652 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:43,652 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,654 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:43,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:43,654 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:43,654 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,654 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:43,670 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,670 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:43,670 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:43,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:43,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:43,670 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:43,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:43,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:43,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:43,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:43,676 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:43,676 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:43,676 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:43,676 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:43,676 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:43,677 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,677 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,677 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:43,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:43,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:43,678 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:43,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,681 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:43,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,686 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:43,686 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,686 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:43,686 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:43,686 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:43,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:43,686 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:43,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:43,686 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,687 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,687 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,687 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:43,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:43,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:43,687 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:43,687 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:43,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:43,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:43,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:43,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195403686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:43,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195403720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:43,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195403721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,728 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:43,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195403723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:43,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195403730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,788 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:43,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:43,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:43,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:43,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:43,793 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:43,793 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:43,793 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:43,793 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:43,794 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,794 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:43,800 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,800 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:43,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:43,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:43,800 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,800 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:43,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:43,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:43,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:43,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:43,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:43,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:43,804 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:43,804 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:43,804 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:43,804 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:43,804 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:43,804 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:43,804 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:43,804 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:43,804 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:43,804 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:43,804 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:43,804 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,805 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:43,805 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:43,806 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,806 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:43,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:43,806 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,806 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:43,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:43,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6390df6e[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:43,808 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:43,808 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,809 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,809 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:43,809 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,809 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:43,809 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:43,810 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:43,813 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,813 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:43,813 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:43,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:43,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:43,813 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:43,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:43,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:43,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:43,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:43,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:43,815 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:43,815 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:43,815 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:43,815 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:43,815 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:43,818 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,818 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:43,818 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:43,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:43,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:43,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:43,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,819 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:43,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:43,821 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,823 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,823 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:43,823 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:43,823 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:43,823 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:43,823 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:43,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:43,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:43,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:43,823 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:43,824 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,824 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,824 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,824 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:43,824 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:43,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:43,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:43,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195403892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,920 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/2e0fccbb235945bc82ac48ba455b65e6 2023-03-30 16:55:43,924 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:43,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:43,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:43,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:43,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:43,927 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:43,927 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:43,927 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:43,927 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:43,928 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,928 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:43,930 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:43,930 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:43,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:43,930 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,930 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:43,930 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/9a30d6f5b06c4bed837c13b0b367f508 2023-03-30 16:55:43,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:43,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:43,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:43,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:43,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:43,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:43,932 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:43,932 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:43,932 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:43,933 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:43,933 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:43,933 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:43,933 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,933 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:43,933 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:43,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:43,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195403928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:43,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195403929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,934 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,934 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:43,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:43,935 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:43,935 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:43,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:43,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@413df325[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:43,937 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:43,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:43,937 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195403929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,939 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,939 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:43,939 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,939 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:43,939 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:43,939 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:43,941 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:43,941 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:43,941 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,941 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:43,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:43,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:43,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:43,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:43,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:43,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:43,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:43,943 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:43,943 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:43,943 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:43,943 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:43,943 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:43,945 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,945 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:43,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:43,945 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:43,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,945 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:43,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:43,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:43,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:43,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195403939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:43,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,947 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:43,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:43,951 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,951 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:43,951 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:43,951 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:43,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:43,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:43,951 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:43,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:43,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:43,951 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:43,951 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:43,951 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:43,951 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,951 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:43,951 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:43,951 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:43,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:43,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:43,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:44,051 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:44,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:44,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:44,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:44,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:44,055 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:44,055 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:44,055 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:44,055 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:44,055 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,055 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,057 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,057 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:44,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:44,058 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,058 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:44,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:44,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:44,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:44,059 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:44,059 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:44,059 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:44,059 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:44,059 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:44,059 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:44,059 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:44,059 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:44,059 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:44,059 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:44,059 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:44,059 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,060 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:44,060 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:44,061 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,061 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,061 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,061 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:44,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:44,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b739556[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:44,062 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:44,062 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,063 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,064 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:44,064 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:44,064 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:44,064 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,064 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:44,065 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,065 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:44,065 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:44,065 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:44,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:44,068 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:44,068 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:44,068 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:44,068 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:44,068 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:44,070 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,070 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,070 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:44,070 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,073 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,075 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:44,075 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:44,075 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:44,076 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:44,075 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,076 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:44,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:44,076 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,076 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,076 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,076 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,076 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:44,076 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:44,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:44,176 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:44,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:44,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:44,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:44,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:44,179 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:44,179 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:44,179 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:44,179 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:44,180 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,180 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,181 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,181 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:44,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:44,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:44,181 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,181 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:44,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:44,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:44,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:44,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:44,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:44,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:44,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:44,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:44,182 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:44,183 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:44,183 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:44,183 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:44,183 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:44,183 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:44,183 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,183 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:44,183 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:44,188 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,188 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,189 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,189 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:44,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2629b187[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:44,191 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:44,191 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,192 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,192 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:44,192 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:44,192 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,192 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:44,192 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:44,193 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:44,193 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:44,193 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,193 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:44,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:44,195 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:44,195 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:44,195 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:44,195 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:44,195 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:44,197 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,198 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,198 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,198 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:44,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,199 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,202 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,202 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,202 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:44,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:44,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:44,202 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:44,203 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:44,203 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:44,202 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,202 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:44,203 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:44,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:44,203 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:44,203 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,203 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:44,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:44,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:44,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195404200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:44,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195404235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:44,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195404236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:44,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195404239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:44,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195404248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,303 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:44,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:44,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:44,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:44,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:44,306 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:44,306 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:44,306 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:44,306 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:44,310 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,310 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,311 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:44,311 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:44,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:44,311 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,311 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:44,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:44,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:44,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:44,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:44,312 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:44,313 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:44,313 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:44,313 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:44,313 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:44,313 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:44,313 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:44,313 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:44,313 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:44,313 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:44,313 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:44,313 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,313 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:44,313 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:44,314 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,314 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,315 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,315 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:44,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:44,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43277183[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:44,316 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:44,316 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,318 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,318 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,318 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:44,318 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:44,318 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:44,318 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:44,324 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,324 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:44,324 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:44,324 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:44,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:44,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:44,334 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:44,334 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:44,334 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:44,334 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:44,334 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:44,338 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,338 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,338 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:44,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,338 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:44,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:44,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,340 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,341 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/8cec52281b3e4f31b63029f9432f4f83 2023-03-30 16:55:44,342 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,342 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,342 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,342 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:44,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:44,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:44,342 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:44,343 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:44,343 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:44,342 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,343 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,343 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:44,343 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:44,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:44,343 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:44,347 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/2e0fccbb235945bc82ac48ba455b65e6 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/2e0fccbb235945bc82ac48ba455b65e6 2023-03-30 16:55:44,351 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/2e0fccbb235945bc82ac48ba455b65e6, entries=150, sequenceid=116, filesize=11.6 K 2023-03-30 16:55:44,351 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/9a30d6f5b06c4bed837c13b0b367f508 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/9a30d6f5b06c4bed837c13b0b367f508 2023-03-30 16:55:44,369 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/9a30d6f5b06c4bed837c13b0b367f508, entries=150, sequenceid=116, filesize=11.6 K 2023-03-30 16:55:44,373 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/8cec52281b3e4f31b63029f9432f4f83 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/8cec52281b3e4f31b63029f9432f4f83 2023-03-30 16:55:44,377 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/8cec52281b3e4f31b63029f9432f4f83, entries=150, sequenceid=116, filesize=11.6 K 2023-03-30 16:55:44,377 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~106.03 KB/108576, currentSize=147.60 KB/151140 for 4038776cda6db2eaa29896eab9262a79 in 869ms, sequenceid=116, compaction requested=true 2023-03-30 16:55:44,378 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:44,378 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:44,378 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:55:44,378 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:55:44,378 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:44,378 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:44,381 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:55:44,381 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/A is initiating minor compaction (all files) 2023-03-30 16:55:44,381 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/A in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:44,381 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/a35125cf1c424f879d9640edaab88071, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/bed124d544914fdb9ee870765f1048dc, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/1d529e778f9a40f08f49bce3609b0710, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/2e0fccbb235945bc82ac48ba455b65e6] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=46.5 K 2023-03-30 16:55:44,383 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:55:44,383 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting a35125cf1c424f879d9640edaab88071, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680195340756 2023-03-30 16:55:44,383 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/B is initiating minor compaction (all files) 2023-03-30 16:55:44,383 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/B in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:44,383 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/5685f49a3f82405bad04127e8d9985ac, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/6ba3d430cfe3496a9edddc1600c24892, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/74e38e865078432ca2578c1454cba9ad, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/9a30d6f5b06c4bed837c13b0b367f508] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=46.5 K 2023-03-30 16:55:44,383 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting bed124d544914fdb9ee870765f1048dc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1680195341117 2023-03-30 16:55:44,383 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 5685f49a3f82405bad04127e8d9985ac, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680195340756 2023-03-30 16:55:44,383 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 1d529e778f9a40f08f49bce3609b0710, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1680195342253 2023-03-30 16:55:44,384 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 6ba3d430cfe3496a9edddc1600c24892, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1680195341117 2023-03-30 16:55:44,384 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 2e0fccbb235945bc82ac48ba455b65e6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680195343401 2023-03-30 16:55:44,384 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 74e38e865078432ca2578c1454cba9ad, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1680195342253 2023-03-30 16:55:44,385 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 9a30d6f5b06c4bed837c13b0b367f508, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680195343401 2023-03-30 16:55:44,393 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#A#compaction#371 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:44,394 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#B#compaction#372 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:44,408 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/de0bd5ac988b4a278737c4ec7f9b9022 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/de0bd5ac988b4a278737c4ec7f9b9022 2023-03-30 16:55:44,414 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/A of 4038776cda6db2eaa29896eab9262a79 into de0bd5ac988b4a278737c4ec7f9b9022(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:44,414 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:44,414 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/A, priority=12, startTime=1680195344378; duration=0sec 2023-03-30 16:55:44,414 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:44,414 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:55:44,422 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:55:44,422 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/C is initiating minor compaction (all files) 2023-03-30 16:55:44,422 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/C in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:44,422 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ad3417c1cb274c6699ea42f4a1608257, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/59500937e31646708ad766f6fd36523c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/7b40eaa9ae374f4b9612f7d9067a4c75, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/8cec52281b3e4f31b63029f9432f4f83] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=46.5 K 2023-03-30 16:55:44,423 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting ad3417c1cb274c6699ea42f4a1608257, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680195340756 2023-03-30 16:55:44,423 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 59500937e31646708ad766f6fd36523c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1680195341117 2023-03-30 16:55:44,424 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 7b40eaa9ae374f4b9612f7d9067a4c75, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1680195342253 2023-03-30 16:55:44,424 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 8cec52281b3e4f31b63029f9432f4f83, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680195343401 2023-03-30 16:55:44,438 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#C#compaction#373 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:44,443 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:44,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:44,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:44,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:44,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:44,448 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:44,448 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:44,448 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:44,448 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:44,448 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,448 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,449 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,449 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:44,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:44,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:44,450 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,450 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:44,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:44,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:44,451 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/8b87ce28138740d292f0f61659dd8efd as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/8b87ce28138740d292f0f61659dd8efd 2023-03-30 16:55:44,458 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/C of 4038776cda6db2eaa29896eab9262a79 into 8b87ce28138740d292f0f61659dd8efd(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:44,459 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:44,459 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/C, priority=12, startTime=1680195344378; duration=0sec 2023-03-30 16:55:44,459 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:44,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:44,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:44,460 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:44,460 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:44,460 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:44,460 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:44,460 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:44,460 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:44,460 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:44,460 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=147.60 KB heapSize=414.61 KB 2023-03-30 16:55:44,461 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:44,461 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:44,461 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:44,461 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:44,461 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:44,461 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:44,471 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/b596abb4179b45b3ac5d03d127113c3e 2023-03-30 16:55:44,505 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/305ed549fa09479ea6f1e5838a3411a9 2023-03-30 16:55:44,542 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/83f05324ea674aef8dc84e0d7c67c76f 2023-03-30 16:55:44,546 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/b596abb4179b45b3ac5d03d127113c3e as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/b596abb4179b45b3ac5d03d127113c3e 2023-03-30 16:55:44,549 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/b596abb4179b45b3ac5d03d127113c3e, entries=150, sequenceid=172, filesize=11.8 K 2023-03-30 16:55:44,550 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/305ed549fa09479ea6f1e5838a3411a9 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/305ed549fa09479ea6f1e5838a3411a9 2023-03-30 16:55:44,558 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/305ed549fa09479ea6f1e5838a3411a9, entries=150, sequenceid=172, filesize=11.8 K 2023-03-30 16:55:44,559 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/83f05324ea674aef8dc84e0d7c67c76f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/83f05324ea674aef8dc84e0d7c67c76f 2023-03-30 16:55:44,564 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/83f05324ea674aef8dc84e0d7c67c76f, entries=150, sequenceid=172, filesize=11.8 K 2023-03-30 16:55:44,565 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~414.56 KB/424512, currentSize=0 B/0 for 4038776cda6db2eaa29896eab9262a79 in 105ms, sequenceid=172, compaction requested=false 2023-03-30 16:55:44,565 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:44,565 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:44,565 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:44,565 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:44,565 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,565 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:44,565 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:44,568 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,568 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,569 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,569 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:44,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:44,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dad0619[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:44,570 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:44,571 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,574 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,574 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:44,574 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:44,574 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:44,574 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,574 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:44,577 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:44,577 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:44,577 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,577 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:44,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:44,582 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:44,582 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:44,582 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:44,582 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:44,582 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:44,588 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,588 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:44,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,589 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:44,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:44,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,590 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,591 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,595 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:44,595 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:44,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:44,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:44,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:44,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:44,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:44,596 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,596 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,596 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,596 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,596 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,596 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,596 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:44,596 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:44,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:44,597 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:44,597 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:44,697 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:44,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:44,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:44,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:44,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:44,701 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:44,701 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:44,701 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:44,701 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:44,701 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,701 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,703 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:44,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:44,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:44,703 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,703 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,703 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:44,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:44,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:44,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:44,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:44,704 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:44,704 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:44,704 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:44,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:44,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:44,705 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:44,705 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:44,708 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:44,708 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:44,708 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:44,708 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:44,708 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,708 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:44,708 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:44,710 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,710 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,711 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,711 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:44,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:44,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ae3f7df[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:44,712 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:44,712 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,713 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,713 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:44,713 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:44,713 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:44,713 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,714 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:44,718 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,718 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:44,718 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:44,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,718 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:44,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:44,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:44,720 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:44,720 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:44,720 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:44,720 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:44,720 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:44,722 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,722 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:44,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,722 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:44,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:44,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,723 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,724 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,726 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,726 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,726 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,726 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:44,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:44,726 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:44,726 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:44,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:44,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:44,726 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:44,727 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:44,727 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:44,726 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,727 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,727 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:44,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:44,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:44,732 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:55:44,732 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:44,732 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:44,732 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:44,732 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:44,732 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:44,733 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:44,740 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/7e5e451433864e049670c113afa2a5f5 2023-03-30 16:55:44,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:44,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195404790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195344755, max timestamp=1680195344802Num uniques 859; 2023-03-30 16:55:44,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:55:44,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195344755, max timestamp=1680195344802Num uniques 859; 2023-03-30 16:55:44,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:55:44,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195344755, max timestamp=1680195344802Num uniques 859; 2023-03-30 16:55:44,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:55:44,827 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/94ef97fac59144d0ade6d1f7bbe22b40 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/94ef97fac59144d0ade6d1f7bbe22b40 2023-03-30 16:55:44,827 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:44,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:44,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:44,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:44,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:44,830 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:44,830 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:44,830 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:44,830 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:44,830 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,830 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,831 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/B of 4038776cda6db2eaa29896eab9262a79 into 94ef97fac59144d0ade6d1f7bbe22b40(size=11.8 K), total size for store is 23.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:44,831 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:44,831 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/B, priority=12, startTime=1680195344378; duration=0sec 2023-03-30 16:55:44,831 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:44,832 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,832 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:44,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:44,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:44,832 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,832 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:44,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:44,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:44,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:44,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:44,833 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:44,833 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:44,833 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:44,833 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:44,833 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:44,833 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:44,833 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:44,833 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:44,834 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:44,834 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:44,834 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:44,834 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,834 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:44,834 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:44,836 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,836 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,837 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,837 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:44,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@112dc48d[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:44,838 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:44,838 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,839 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,839 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:44,839 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:44,839 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:44,839 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,839 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:44,841 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,841 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:44,841 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:44,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,841 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:44,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:44,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195404835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:44,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:44,845 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:44,845 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:44,845 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:44,845 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:44,845 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:44,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:44,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195404835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:44,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195404836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:44,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195404837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,847 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,847 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,847 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,848 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:44,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,849 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,852 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:44,852 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:44,852 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:44,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:44,852 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,853 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:44,853 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:44,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:44,853 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,853 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,853 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,853 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,853 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:44,853 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:44,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:44,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:44,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:44,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195404905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:44,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195404944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:44,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195404947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:44,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195404948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:44,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195404948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,953 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:44,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:44,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:44,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:44,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:44,956 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:44,956 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:44,956 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:44,956 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:44,956 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,956 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,958 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,958 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:44,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:44,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:44,958 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,958 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:44,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:44,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:44,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:44,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:44,959 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:44,959 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:44,959 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:44,959 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:44,959 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:44,959 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:44,959 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:44,959 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:44,960 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:44,960 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:44,960 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:44,960 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,960 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:44,960 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:44,961 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,961 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,962 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,962 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:44,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:44,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69fe1442[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:44,963 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:44,963 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,964 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,964 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:44,964 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:44,964 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:44,964 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,964 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:44,967 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,967 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:44,967 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:44,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,967 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:44,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:44,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:44,969 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:44,969 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:44,969 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:44,969 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:44,969 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:44,971 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,971 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:44,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,971 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:44,971 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:44,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:44,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:44,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:44,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,972 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:44,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:44,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,975 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:44,975 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:44,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:44,975 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:44,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:44,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:44,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:44,975 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,975 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,975 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:44,975 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:44,975 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:44,975 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:44,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:44,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:44,975 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,975 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:44,975 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,075 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:45,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:45,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:45,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:45,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:45,078 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:45,079 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:45,079 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:45,079 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:45,079 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,079 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:45,080 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:45,080 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:45,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:45,081 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:45,081 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:45,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:45,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:45,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:45,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:45,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:45,082 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:45,082 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:45,082 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:45,082 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:45,082 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:45,082 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:45,082 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:45,082 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:45,082 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:45,082 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:45,082 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:45,082 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,082 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:45,082 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:45,084 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,084 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:45,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:45,085 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,085 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:45,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:45,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:45,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:45,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:45,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:45,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c665aaa[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:45,086 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:45,086 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,087 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,087 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:45,087 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:45,087 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:45,088 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,088 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:45,088 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,088 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:45,089 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:45,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:45,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:45,089 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:45,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:45,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:45,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:45,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:45,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:45,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:45,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:45,091 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:45,091 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:45,091 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:45,091 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:45,091 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:45,096 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,096 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:45,096 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:45,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:45,097 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:45,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:45,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,098 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:45,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:45,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,100 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,100 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:45,100 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:45,100 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,100 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,100 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,100 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,100 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:45,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:45,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:45,100 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:45,101 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:45,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:45,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:45,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:45,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:45,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:45,101 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:45,101 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:45,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:45,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195405109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:45,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195405149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,153 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/e756af1ce0c14e2d9f8cb698daa92518 2023-03-30 16:55:45,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:45,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195405154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:45,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195405154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:45,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195405154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,163 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/fa3b499595084e33bec03085e683ad2a 2023-03-30 16:55:45,166 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/7e5e451433864e049670c113afa2a5f5 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/7e5e451433864e049670c113afa2a5f5 2023-03-30 16:55:45,169 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/7e5e451433864e049670c113afa2a5f5, entries=150, sequenceid=183, filesize=11.8 K 2023-03-30 16:55:45,170 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/e756af1ce0c14e2d9f8cb698daa92518 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/e756af1ce0c14e2d9f8cb698daa92518 2023-03-30 16:55:45,173 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/e756af1ce0c14e2d9f8cb698daa92518, entries=100, sequenceid=183, filesize=9.4 K 2023-03-30 16:55:45,174 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/fa3b499595084e33bec03085e683ad2a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/fa3b499595084e33bec03085e683ad2a 2023-03-30 16:55:45,177 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/fa3b499595084e33bec03085e683ad2a, entries=100, sequenceid=183, filesize=9.4 K 2023-03-30 16:55:45,178 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 4038776cda6db2eaa29896eab9262a79 in 446ms, sequenceid=183, compaction requested=true 2023-03-30 16:55:45,178 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:45,178 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:45,178 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:45,178 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:55:45,178 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:55:45,178 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:45,180 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33804 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:55:45,180 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36194 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:55:45,180 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/B is initiating minor compaction (all files) 2023-03-30 16:55:45,180 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/A is initiating minor compaction (all files) 2023-03-30 16:55:45,180 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/B in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:45,180 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/A in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:45,180 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/94ef97fac59144d0ade6d1f7bbe22b40, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/305ed549fa09479ea6f1e5838a3411a9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/e756af1ce0c14e2d9f8cb698daa92518] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=33.0 K 2023-03-30 16:55:45,180 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/de0bd5ac988b4a278737c4ec7f9b9022, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/b596abb4179b45b3ac5d03d127113c3e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/7e5e451433864e049670c113afa2a5f5] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=35.3 K 2023-03-30 16:55:45,181 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 94ef97fac59144d0ade6d1f7bbe22b40, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680195343401 2023-03-30 16:55:45,181 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting de0bd5ac988b4a278737c4ec7f9b9022, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680195343401 2023-03-30 16:55:45,181 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 305ed549fa09479ea6f1e5838a3411a9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680195343611 2023-03-30 16:55:45,181 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting b596abb4179b45b3ac5d03d127113c3e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680195343611 2023-03-30 16:55:45,182 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting e756af1ce0c14e2d9f8cb698daa92518, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1680195344713 2023-03-30 16:55:45,182 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 7e5e451433864e049670c113afa2a5f5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1680195344713 2023-03-30 16:55:45,191 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#B#compaction#380 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:45,191 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#A#compaction#381 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:45,201 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:45,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:45,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:45,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:45,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:45,211 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:45,212 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:45,212 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:45,212 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:45,212 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,212 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:45,213 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:45,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:45,213 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:45,214 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,214 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:45,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:45,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:45,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:45,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:45,215 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:45,215 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:45,215 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:45,221 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:45,221 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:45,221 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:45,221 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:45,221 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-30 16:55:45,222 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:45,222 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:45,222 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:45,222 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:45,222 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:45,222 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:45,232 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/4ac7e696e31b4c34807cd1eeb7e323cb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/4ac7e696e31b4c34807cd1eeb7e323cb 2023-03-30 16:55:45,236 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/B of 4038776cda6db2eaa29896eab9262a79 into 4ac7e696e31b4c34807cd1eeb7e323cb(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:45,236 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:45,236 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/B, priority=13, startTime=1680195345178; duration=0sec 2023-03-30 16:55:45,236 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:45,236 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:55:45,237 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33804 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:55:45,237 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/C is initiating minor compaction (all files) 2023-03-30 16:55:45,238 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/C in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:45,238 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/8b87ce28138740d292f0f61659dd8efd, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/83f05324ea674aef8dc84e0d7c67c76f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/fa3b499595084e33bec03085e683ad2a] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=33.0 K 2023-03-30 16:55:45,238 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 8b87ce28138740d292f0f61659dd8efd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1680195343401 2023-03-30 16:55:45,241 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 83f05324ea674aef8dc84e0d7c67c76f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680195343611 2023-03-30 16:55:45,242 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting fa3b499595084e33bec03085e683ad2a, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1680195344713 2023-03-30 16:55:45,250 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#C#compaction#383 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:45,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:45,418 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:45,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:45,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195405444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:45,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195405456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:45,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195405456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:45,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195405456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:45,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195405457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:45,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195405548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,621 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/cc765ee7d6fe423da2e20cc574584132 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/cc765ee7d6fe423da2e20cc574584132 2023-03-30 16:55:45,625 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/A of 4038776cda6db2eaa29896eab9262a79 into cc765ee7d6fe423da2e20cc574584132(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:45,625 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:45,625 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/A, priority=13, startTime=1680195345178; duration=0sec 2023-03-30 16:55:45,625 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:45,637 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/3810673bf94e46458cbb5a9237030ccb 2023-03-30 16:55:45,647 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/f4610c52a8474e31b888220c2a6011ad 2023-03-30 16:55:45,659 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/7ec7546c7fde4cd199d9cec84e79eda3 2023-03-30 16:55:45,665 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/3810673bf94e46458cbb5a9237030ccb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/3810673bf94e46458cbb5a9237030ccb 2023-03-30 16:55:45,666 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/73dc51beadc04ef8a7004ebf41001e36 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/73dc51beadc04ef8a7004ebf41001e36 2023-03-30 16:55:45,674 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/3810673bf94e46458cbb5a9237030ccb, entries=150, sequenceid=220, filesize=11.8 K 2023-03-30 16:55:45,674 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/C of 4038776cda6db2eaa29896eab9262a79 into 73dc51beadc04ef8a7004ebf41001e36(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:45,674 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:45,674 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/C, priority=13, startTime=1680195345178; duration=0sec 2023-03-30 16:55:45,674 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:45,675 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/f4610c52a8474e31b888220c2a6011ad as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/f4610c52a8474e31b888220c2a6011ad 2023-03-30 16:55:45,679 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/f4610c52a8474e31b888220c2a6011ad, entries=150, sequenceid=220, filesize=11.8 K 2023-03-30 16:55:45,682 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/7ec7546c7fde4cd199d9cec84e79eda3 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/7ec7546c7fde4cd199d9cec84e79eda3 2023-03-30 16:55:45,688 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/7ec7546c7fde4cd199d9cec84e79eda3, entries=150, sequenceid=220, filesize=11.8 K 2023-03-30 16:55:45,688 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 4038776cda6db2eaa29896eab9262a79 in 467ms, sequenceid=220, compaction requested=false 2023-03-30 16:55:45,688 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:45,688 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:45,689 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:45,689 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:45,689 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,692 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:45,692 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:45,697 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,697 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:45,697 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,697 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9de15f6[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:45,699 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:45,700 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,701 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,701 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:45,702 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,702 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:45,702 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:45,702 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:45,704 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,704 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:45,704 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:45,704 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:45,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:45,708 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:45,708 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:45,708 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:45,708 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:45,708 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:45,713 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,713 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:45,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,713 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:45,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:45,713 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:45,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:45,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:45,717 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:45,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:45,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,719 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,719 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:45,719 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:45,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:45,719 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,719 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,719 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,719 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,719 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:45,719 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:45,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:45,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:45,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:45,719 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:45,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:45,720 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:45,720 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:45,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:45,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:45,754 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 16:55:45,755 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:45,755 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:45,755 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:45,755 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:45,755 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:45,755 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:45,761 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/7f59055f532a48308226c63f879d6997 2023-03-30 16:55:45,773 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/6a49280309724c66978fcfabbe57d849 2023-03-30 16:55:45,781 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/c00095d3f5524344951d1fde496bee3b 2023-03-30 16:55:45,784 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/7f59055f532a48308226c63f879d6997 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/7f59055f532a48308226c63f879d6997 2023-03-30 16:55:45,787 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/7f59055f532a48308226c63f879d6997, entries=200, sequenceid=234, filesize=14.1 K 2023-03-30 16:55:45,788 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/6a49280309724c66978fcfabbe57d849 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/6a49280309724c66978fcfabbe57d849 2023-03-30 16:55:45,791 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/6a49280309724c66978fcfabbe57d849, entries=150, sequenceid=234, filesize=11.8 K 2023-03-30 16:55:45,795 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/c00095d3f5524344951d1fde496bee3b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/c00095d3f5524344951d1fde496bee3b 2023-03-30 16:55:45,801 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/c00095d3f5524344951d1fde496bee3b, entries=150, sequenceid=234, filesize=11.8 K 2023-03-30 16:55:45,802 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=53.67 KB/54960 for 4038776cda6db2eaa29896eab9262a79 in 48ms, sequenceid=234, compaction requested=true 2023-03-30 16:55:45,802 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:45,802 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:45,802 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:55:45,802 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:55:45,802 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:45,802 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:45,803 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38840 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:55:45,803 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/A is initiating minor compaction (all files) 2023-03-30 16:55:45,803 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:55:45,803 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/A in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:45,803 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/B is initiating minor compaction (all files) 2023-03-30 16:55:45,803 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/cc765ee7d6fe423da2e20cc574584132, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/3810673bf94e46458cbb5a9237030ccb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/7f59055f532a48308226c63f879d6997] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=37.9 K 2023-03-30 16:55:45,803 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/B in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:45,803 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/4ac7e696e31b4c34807cd1eeb7e323cb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/f4610c52a8474e31b888220c2a6011ad, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/6a49280309724c66978fcfabbe57d849] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=35.6 K 2023-03-30 16:55:45,803 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting cc765ee7d6fe423da2e20cc574584132, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1680195343617 2023-03-30 16:55:45,804 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 4ac7e696e31b4c34807cd1eeb7e323cb, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1680195343617 2023-03-30 16:55:45,804 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 3810673bf94e46458cbb5a9237030ccb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680195344820 2023-03-30 16:55:45,804 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting f4610c52a8474e31b888220c2a6011ad, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680195344820 2023-03-30 16:55:45,804 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 7f59055f532a48308226c63f879d6997, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1680195345434 2023-03-30 16:55:45,804 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 6a49280309724c66978fcfabbe57d849, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1680195345434 2023-03-30 16:55:45,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:45,805 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-30 16:55:45,805 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:45,805 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:45,805 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:45,805 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:45,805 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:45,805 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:45,821 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:45,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:45,822 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#A#compaction#390 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:45,823 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#B#compaction#391 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:45,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:45,836 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/5443d9ece19e4a138c2655229ec41bc8 2023-03-30 16:55:45,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:45,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:45,841 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:45,841 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:45,841 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:45,841 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:45,841 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,841 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:45,842 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:45,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:45,842 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:45,843 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,843 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:45,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:45,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:45,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:45,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:45,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:45,852 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:45,852 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:45,852 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:45,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:45,868 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:45,868 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:45,869 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:45,869 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:45,869 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:45,869 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:45,869 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:45,869 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,869 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:45,869 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:45,871 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:45,871 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:45,872 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,872 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:45,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:45,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:45,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:45,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:45,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7628b1ec[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:45,873 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:45,873 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,875 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/eb285f1e7b724a0dace42da0199d18c6 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/eb285f1e7b724a0dace42da0199d18c6 2023-03-30 16:55:45,875 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,875 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:45,876 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:45,876 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,876 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:45,876 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:45,878 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:45,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:45,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:45,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:45,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,881 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/A of 4038776cda6db2eaa29896eab9262a79 into eb285f1e7b724a0dace42da0199d18c6(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:45,881 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:45,881 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/A, priority=13, startTime=1680195345802; duration=0sec 2023-03-30 16:55:45,881 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:45,881 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:55:45,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:45,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:45,882 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:45,882 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:45,882 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:45,882 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:45,882 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:45,882 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:55:45,882 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/C is initiating minor compaction (all files) 2023-03-30 16:55:45,882 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/C in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:45,882 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/73dc51beadc04ef8a7004ebf41001e36, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/7ec7546c7fde4cd199d9cec84e79eda3, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/c00095d3f5524344951d1fde496bee3b] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=35.6 K 2023-03-30 16:55:45,883 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 73dc51beadc04ef8a7004ebf41001e36, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1680195343617 2023-03-30 16:55:45,883 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 7ec7546c7fde4cd199d9cec84e79eda3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680195344820 2023-03-30 16:55:45,883 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting c00095d3f5524344951d1fde496bee3b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1680195345434 2023-03-30 16:55:45,883 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,884 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:45,884 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,884 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:45,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,886 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:45,888 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:45,888 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,888 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:45,889 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:45,888 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:45,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:45,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:45,889 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:45,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:45,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:45,889 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:45,889 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,889 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,889 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:45,889 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,889 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:45,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:45,890 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#C#compaction#393 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:45,910 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/3d51ffd9dc9c41e8afb92f8545a000be as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/3d51ffd9dc9c41e8afb92f8545a000be 2023-03-30 16:55:45,915 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/C of 4038776cda6db2eaa29896eab9262a79 into 3d51ffd9dc9c41e8afb92f8545a000be(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:45,915 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:45,915 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/C, priority=13, startTime=1680195345802; duration=0sec 2023-03-30 16:55:45,915 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:45,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:45,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195405942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:45,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195405960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:45,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195405963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:45,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195405963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:45,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195405963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,989 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:45,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:45,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:45,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:45,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:45,992 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:45,992 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:45,992 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:45,992 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:45,994 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,994 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:45,997 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,997 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:45,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:45,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:45,997 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,997 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:45,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:45,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:45,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:45,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:45,998 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:45,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:45,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:45,998 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:45,998 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:45,998 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:45,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:45,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:45,999 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:45,999 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:45,999 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:45,999 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:45,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:45,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:45,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:45,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:46,000 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:46,001 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,001 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:46,001 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,001 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:46,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:46,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f6df76f[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:46,003 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:46,003 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,004 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,004 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:46,004 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:46,004 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:46,004 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,004 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:46,005 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:46,005 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:46,005 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:46,006 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:46,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:46,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:46,009 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:46,009 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:46,009 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:46,009 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:46,009 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:46,013 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,013 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:46,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,015 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,016 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:46,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,018 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,022 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,022 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,022 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,022 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:46,022 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:46,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:46,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:46,022 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:46,022 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:46,023 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,023 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:46,023 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:46,023 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,023 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:46,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:46,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195406047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,123 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:46,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:46,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:46,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:46,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:46,126 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:46,126 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:46,126 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:46,126 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:46,126 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,126 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,127 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:46,127 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:46,128 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,128 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:46,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:46,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:46,129 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:46,129 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:46,129 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:46,129 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:46,129 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:46,129 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:46,129 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:46,129 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:46,130 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:46,130 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:46,130 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:46,130 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,130 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:46,130 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:46,132 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,132 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:46,132 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,132 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:46,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:46,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31005322[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:46,134 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:46,134 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,135 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,135 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:46,135 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:46,135 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,135 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:46,135 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:46,136 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,136 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:46,136 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:46,137 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:46,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:46,139 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:46,139 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:46,139 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:46,139 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:46,139 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:46,140 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,140 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,140 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:46,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:46,141 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:46,142 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,145 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,145 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:46,145 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:46,145 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,145 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:46,146 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:46,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:46,145 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:46,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:46,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:46,146 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,146 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,146 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,146 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:46,146 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:46,246 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:46,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:46,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:46,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:46,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:46,250 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:46,250 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:46,250 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:46,250 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:46,250 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,250 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,251 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:46,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:46,251 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,252 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,252 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:46,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:46,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:46,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:46,253 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:46,253 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:46,253 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:46,253 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:46,253 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:46,253 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:46,253 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:46,254 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:46,254 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:46,254 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:46,254 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:46,254 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,254 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:46,254 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:46,255 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,255 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:46,256 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,256 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:46,257 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/48035ff5d48f4cf7b99f995e9b8106eb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/48035ff5d48f4cf7b99f995e9b8106eb 2023-03-30 16:55:46,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:46,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:46,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:46,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195406254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:46,261 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/B of 4038776cda6db2eaa29896eab9262a79 into 48035ff5d48f4cf7b99f995e9b8106eb(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:46,261 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:46,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:46,261 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/B, priority=13, startTime=1680195345802; duration=0sec 2023-03-30 16:55:46,261 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:46,261 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,261 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:46,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48888151[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:46,262 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,262 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:46,263 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:46,263 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:46,263 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,263 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:46,265 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,265 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:46,265 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:46,265 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:46,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:46,267 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:46,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:46,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:46,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:46,267 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:46,269 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,269 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,269 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:46,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:46,269 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,270 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,272 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:46,272 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:46,272 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:46,272 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:46,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:46,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:46,273 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:46,273 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:46,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:46,272 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,273 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,273 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,273 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,273 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,273 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:46,275 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/385a6f9105f648ffae0e4bff001a1b70 2023-03-30 16:55:46,373 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:46,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:46,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:46,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:46,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:46,376 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:46,376 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:46,376 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:46,376 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:46,377 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,377 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,378 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:46,378 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:46,379 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,379 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:46,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:46,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:46,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:46,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:46,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:46,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:46,380 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:46,380 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:46,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:46,380 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:46,380 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:46,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:46,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:46,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:46,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:46,382 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,382 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:46,383 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,383 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:46,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:46,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b9cee7d[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:46,384 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:46,384 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,385 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,385 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:46,385 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:46,385 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:46,386 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,386 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:46,386 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,387 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:46,387 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:46,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:46,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:46,387 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:46,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:46,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:46,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:46,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:46,389 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:46,389 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:46,389 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:46,389 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:46,389 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:46,390 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,390 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:46,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,390 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:46,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,391 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,392 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,395 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,395 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,395 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,395 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,395 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,395 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,395 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:46,395 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:46,395 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:46,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:46,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:46,395 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:46,396 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:46,396 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:46,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:46,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:46,496 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:46,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:46,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:46,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:46,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:46,499 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:46,499 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:46,499 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:46,499 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:46,499 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,499 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,503 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:46,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:46,503 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,504 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,504 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:46,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:46,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:46,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:46,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:46,504 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:46,504 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:46,504 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:46,505 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:46,505 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:46,505 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:46,505 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:46,505 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:46,505 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:46,505 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:46,505 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:46,505 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,505 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:46,505 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,506 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:46,506 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,507 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:46,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:46,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bcab8ee[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:46,508 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:46,508 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,509 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:46,509 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:46,509 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,509 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:46,509 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:46,512 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,512 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:46,512 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:46,512 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:46,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:46,514 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:46,514 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:46,514 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:46,514 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:46,514 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:46,515 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,515 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:46,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,515 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:46,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:46,520 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,521 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,523 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,523 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,523 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,523 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,523 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,523 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,523 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:46,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:46,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:46,523 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:46,523 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:46,524 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:46,524 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:46,524 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:46,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:46,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:46,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195406559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,624 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:46,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:46,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:46,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:46,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:46,627 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:46,627 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:46,627 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:46,627 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:46,628 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,628 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,629 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:46,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:46,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:46,629 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,629 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,629 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:46,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:46,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:46,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:46,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:46,630 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:46,630 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:46,630 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:46,630 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:46,631 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:46,631 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:46,631 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:46,631 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:46,634 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,634 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:46,635 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,635 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:46,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:46,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48607df6[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:46,636 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:46,636 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,637 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,637 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:46,637 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:46,637 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:46,637 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,638 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:46,639 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,639 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:46,639 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:46,639 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:46,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:46,641 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:46,641 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:46,641 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:46,641 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:46,641 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:46,643 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,643 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:46,643 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:46,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:46,643 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,644 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,650 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:46,650 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:46,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:46,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:46,650 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:46,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:46,650 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:46,650 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:46,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:46,650 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,650 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:46,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:46,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:46,650 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,650 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,650 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,650 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:46,651 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,684 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/ae1dcb6aa9024402928a1849b73f2317 2023-03-30 16:55:46,687 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/5443d9ece19e4a138c2655229ec41bc8 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/5443d9ece19e4a138c2655229ec41bc8 2023-03-30 16:55:46,692 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/5443d9ece19e4a138c2655229ec41bc8, entries=100, sequenceid=246, filesize=9.4 K 2023-03-30 16:55:46,693 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/385a6f9105f648ffae0e4bff001a1b70 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/385a6f9105f648ffae0e4bff001a1b70 2023-03-30 16:55:46,698 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/385a6f9105f648ffae0e4bff001a1b70, entries=100, sequenceid=246, filesize=9.4 K 2023-03-30 16:55:46,699 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/ae1dcb6aa9024402928a1849b73f2317 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ae1dcb6aa9024402928a1849b73f2317 2023-03-30 16:55:46,702 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ae1dcb6aa9024402928a1849b73f2317, entries=100, sequenceid=246, filesize=9.4 K 2023-03-30 16:55:46,702 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4038776cda6db2eaa29896eab9262a79 in 897ms, sequenceid=246, compaction requested=false 2023-03-30 16:55:46,702 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:46,750 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:46,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:46,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:46,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:46,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:46,753 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:46,754 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:46,754 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:46,754 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:46,754 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,754 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,755 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:46,755 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:46,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:46,756 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,756 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:46,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:46,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:46,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:46,757 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:46,757 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:46,757 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-30 16:55:46,757 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:46,757 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:46,757 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:46,757 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:46,757 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:46,757 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:46,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:46,972 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:46,987 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:46,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195406985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:46,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195406986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:46,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195406987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:46,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:46,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195406987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195407062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195407088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195407092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195407093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195407094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,165 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/1a65f3e079d14bb9a46dab9a5fad2d82 2023-03-30 16:55:47,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195407292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195407299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195407300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195407302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,575 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/9fe1879d858149e6aede4670b393df75 2023-03-30 16:55:47,585 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/0f83127160924439bf06047b5ad8f033 2023-03-30 16:55:47,588 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/1a65f3e079d14bb9a46dab9a5fad2d82 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/1a65f3e079d14bb9a46dab9a5fad2d82 2023-03-30 16:55:47,593 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/1a65f3e079d14bb9a46dab9a5fad2d82, entries=150, sequenceid=269, filesize=11.9 K 2023-03-30 16:55:47,593 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/9fe1879d858149e6aede4670b393df75 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/9fe1879d858149e6aede4670b393df75 2023-03-30 16:55:47,597 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/9fe1879d858149e6aede4670b393df75, entries=150, sequenceid=269, filesize=11.9 K 2023-03-30 16:55:47,597 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/0f83127160924439bf06047b5ad8f033 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/0f83127160924439bf06047b5ad8f033 2023-03-30 16:55:47,601 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/0f83127160924439bf06047b5ad8f033, entries=150, sequenceid=269, filesize=11.9 K 2023-03-30 16:55:47,601 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 4038776cda6db2eaa29896eab9262a79 in 844ms, sequenceid=269, compaction requested=true 2023-03-30 16:55:47,601 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:47,601 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:47,601 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:47,602 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:47,602 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,602 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:47,602 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:47,603 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:47,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:47,604 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-30 16:55:47,604 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:47,604 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:47,604 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:47,604 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:47,604 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:47,604 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:47,604 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:47,604 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,604 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:47,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:47,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@446eefa2[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:47,606 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:47,606 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,607 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,607 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:47,607 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:47,607 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:47,609 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:47,609 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:47,609 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,609 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:47,609 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:47,610 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:47,610 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:47,610 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:47,610 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:47,610 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:47,610 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:47,611 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,611 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:47,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,611 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:47,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,613 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,614 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:47,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,616 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,616 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:47,616 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:47,616 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:47,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:47,616 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:47,617 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:47,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:47,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:47,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:47,617 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,617 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,617 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,617 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,617 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:47,617 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:47,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:47,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:47,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:47,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195407630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195407655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195407655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195407655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,717 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:47,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:47,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:47,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:47,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:47,720 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:47,720 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:47,720 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:47,720 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:47,720 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,720 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:47,721 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:47,721 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:47,721 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,721 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:47,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:47,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:47,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:47,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:47,722 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:47,722 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:47,722 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:47,723 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:47,723 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:47,723 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:47,723 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:47,723 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:47,723 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:47,723 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:47,723 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:47,723 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,723 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:47,723 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:47,728 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,728 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:47,728 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,728 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:47,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:47,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@581d0e54[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:47,729 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:47,729 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,730 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,730 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:47,730 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,730 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:47,730 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:47,731 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:47,732 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:47,732 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,732 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:47,732 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:47,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:47,734 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:47,734 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:47,734 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:47,734 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:47,734 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:47,735 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,735 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:47,735 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:47,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,736 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,737 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,740 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,740 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:47,740 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:47,740 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:47,740 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:47,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:47,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:47,740 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:47,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:47,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:47,740 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,741 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,741 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:47,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:47,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:47,741 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:47,741 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,741 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195407755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195407763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195407764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195407764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,841 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:47,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:47,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:47,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:47,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:47,844 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:47,844 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:47,844 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:47,844 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:47,845 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,845 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:47,846 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:47,846 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:47,846 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,846 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:47,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:47,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:47,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:47,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:47,847 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:47,847 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:47,847 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:47,848 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:47,848 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:47,848 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:47,848 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:47,848 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:47,848 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:47,848 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:47,848 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:47,848 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,848 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:47,848 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:47,849 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,849 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:47,849 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,850 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:47,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:47,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76bed071[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:47,851 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:47,851 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,852 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,853 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:47,853 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:47,853 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:47,853 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,853 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:47,855 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,855 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:47,855 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:47,855 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:47,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:47,857 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:47,857 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:47,857 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:47,857 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:47,857 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:47,859 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,859 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:47,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,859 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:47,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,860 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,861 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:47,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:47,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:47,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:47,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:47,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:47,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:47,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:47,864 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:47,864 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:47,864 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:47,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195407961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,964 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:47,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:47,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:47,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:47,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:47,967 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:47,967 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:47,967 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:47,967 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:47,968 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,968 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:47,969 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:47,969 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:47,970 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,970 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:47,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:47,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:47,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:47,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:47,971 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:47,971 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:47,971 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:47,971 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:47,974 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,974 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:47,974 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,974 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:47,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:47,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dfa9c9d[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:47,976 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:47,976 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195407972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195407972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,977 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,977 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:47,977 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:47,977 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:47,977 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,977 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:47,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:47,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195407972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,978 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,978 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:47,978 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:47,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:47,978 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:47,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:47,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:47,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:47,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:47,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:47,981 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:47,981 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:47,981 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:47,981 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:47,981 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:47,982 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,982 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:47,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:47,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,982 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:47,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:47,983 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,984 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:47,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:47,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,987 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:47,987 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:47,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:47,987 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:47,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:47,987 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:47,987 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:47,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:47,987 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:47,987 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:47,987 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:47,987 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:47,987 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,987 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:47,987 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,012 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/e35878dea0e7430c8234c61dee9c8603 2023-03-30 16:55:48,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:48,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57264 deadline: 1680195408074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,087 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:48,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:48,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:48,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:48,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:48,090 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:48,090 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:48,090 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:48,090 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:48,091 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,091 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,092 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:48,092 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:48,092 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,092 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:48,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:48,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:48,093 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:48,093 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:48,093 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:48,094 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:48,094 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:48,094 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:48,094 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:48,094 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:48,094 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:48,094 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:48,094 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:48,094 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,094 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:48,094 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:48,095 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,095 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:48,096 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,096 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:48,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:48,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@eda677f[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:48,097 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:48,097 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,098 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,098 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:48,098 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:48,098 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:48,098 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,098 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:48,103 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,103 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:48,103 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:48,103 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:48,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:48,105 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:48,105 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:48,105 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:48,105 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:48,105 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:48,107 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,107 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,107 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:48,107 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:48,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:48,108 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:48,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,110 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,110 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:48,110 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:48,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:48,110 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:48,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:48,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:48,110 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:48,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:48,110 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,111 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:48,111 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:48,111 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:48,111 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,111 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,111 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,211 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:48,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:48,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:48,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:48,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:48,214 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:48,214 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:48,214 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:48,214 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:48,215 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,215 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,216 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:48,216 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:48,216 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,216 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:48,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:48,217 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:48,217 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:48,218 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:48,218 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:48,218 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:48,218 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:48,218 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:48,218 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:48,218 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:48,218 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:48,218 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:48,218 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:48,218 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,218 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:48,218 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:48,221 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,221 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:48,221 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,221 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:48,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:48,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e85b75e[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:48,223 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:48,223 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,224 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,224 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:48,224 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:48,224 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:48,224 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,224 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:48,225 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,225 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:48,226 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:48,226 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:48,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:48,228 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:48,228 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:48,228 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:48,228 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:48,228 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:48,229 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,229 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:48,229 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:48,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:48,230 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,232 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,236 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,236 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,236 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,236 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,236 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,236 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:48,236 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:48,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:48,236 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,236 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:48,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:48,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:48,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:48,236 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:48,236 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:48,236 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:48,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:48,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:48,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:48,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195408265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:48,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195408278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:48,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195408279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:48,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195408280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,337 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:48,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:48,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:48,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:48,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:48,340 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:48,340 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:48,340 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:48,340 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:48,340 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,340 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,341 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,341 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:48,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:48,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:48,341 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,341 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,342 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:48,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:48,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:48,343 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:48,343 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:48,343 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:48,343 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:48,343 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:48,343 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:48,343 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:48,343 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:48,343 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:48,343 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:48,343 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:48,343 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,343 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:48,343 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:48,344 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:48,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:48,345 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,345 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:48,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:48,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:48,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:48,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e38c5a7[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:48,346 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:48,346 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,347 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,347 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:48,348 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:48,348 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:48,348 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,348 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:48,350 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,350 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:48,350 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:48,350 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:48,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:48,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:48,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:48,365 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:48,365 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:48,365 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:48,365 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:48,365 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:48,367 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,367 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,367 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:48,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:48,367 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:48,368 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,373 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,374 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:48,374 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:48,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:48,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:48,374 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:48,374 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,374 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:48,374 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,374 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,374 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,374 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,374 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:48,374 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:48,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:48,422 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/3a7060e7ceba45a893c0933554c843a5 2023-03-30 16:55:48,474 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:48,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:48,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:48,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:48,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:48,478 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:48,478 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:48,478 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:48,478 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:48,479 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,479 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,480 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:48,480 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:48,480 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,480 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:48,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:48,482 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:48,482 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:48,482 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:48,484 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:48,484 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:48,486 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:48,486 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:48,486 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:48,486 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:48,486 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,486 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:48,486 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:48,489 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:48,489 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:48,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:48,497 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,497 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:48,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:48,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@574fa889[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:48,504 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:48,504 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,505 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,505 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,506 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,506 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:48,506 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:48,506 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:48,507 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:48,507 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,507 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:48,507 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:48,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:48,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:48,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:48,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:48,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:48,512 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:48,512 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:48,512 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:48,512 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:48,512 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:48,521 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,521 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,521 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:48,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:48,523 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,527 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,527 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,527 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,527 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:48,527 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:48,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:48,527 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:48,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:48,527 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:48,527 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:48,527 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:48,527 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:48,527 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,528 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:48,628 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:48,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:48,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:48,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:48,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:48,632 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:48,632 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:48,632 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:48,632 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:48,632 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,632 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,633 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,634 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:48,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:48,634 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,634 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:48,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:48,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:48,636 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:48,636 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:48,636 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:48,640 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:48,640 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:48,640 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:48,640 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:48,640 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:48,640 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:48,640 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:48,640 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:48,640 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,640 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:48,640 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:48,642 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:48,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:48,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@af3dc80[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:48,643 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:48,643 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,646 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,646 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:48,646 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,646 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:48,646 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:48,646 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:48,648 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:48,648 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:48,648 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:48,648 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:48,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:48,651 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:48,651 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:48,651 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:48,651 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:48,651 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:48,652 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,652 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:48,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,652 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:48,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,654 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,655 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,658 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,658 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,658 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,658 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,658 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:48,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:48,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:48,658 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:48,658 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:48,658 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:48,658 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,658 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,658 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:48,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:48,659 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:48,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:48,759 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:48,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:48,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:48,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:48,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:48,762 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:48,762 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:48,762 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:48,762 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:48,762 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,762 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,764 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,764 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:48,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:48,764 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,765 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:48,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:48,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:48,765 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:48,765 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:48,766 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:48,766 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:48,766 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:48,766 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:48,766 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:48,766 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:48,766 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:48,766 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:48,766 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:48,766 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,766 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:48,766 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:48,768 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,768 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:48,768 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,768 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:48,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:48,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@195fa5c7[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:48,769 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:48,770 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,771 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,771 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:48,771 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:48,771 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,771 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:48,771 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:48,773 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:48,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:48,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:48,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:48,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:48,775 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:48,775 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:48,775 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:48,775 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:48,775 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:48,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:48,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195408770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,778 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,778 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:48,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,778 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:48,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:48,779 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,781 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,783 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,783 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,783 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:48,783 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:48,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:48,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:48,783 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:48,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:48,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:48,783 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:48,783 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,783 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,783 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:48,783 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,783 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:48,783 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:48,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:48,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:48,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195408788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:48,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195408788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:48,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195408789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,831 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/f11f183a10e44b4b885c8051c798e2a7 2023-03-30 16:55:48,834 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/e35878dea0e7430c8234c61dee9c8603 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/e35878dea0e7430c8234c61dee9c8603 2023-03-30 16:55:48,845 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/e35878dea0e7430c8234c61dee9c8603, entries=200, sequenceid=283, filesize=14.3 K 2023-03-30 16:55:48,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/3a7060e7ceba45a893c0933554c843a5 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/3a7060e7ceba45a893c0933554c843a5 2023-03-30 16:55:48,850 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/3a7060e7ceba45a893c0933554c843a5, entries=150, sequenceid=283, filesize=11.9 K 2023-03-30 16:55:48,851 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/f11f183a10e44b4b885c8051c798e2a7 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/f11f183a10e44b4b885c8051c798e2a7 2023-03-30 16:55:48,861 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/f11f183a10e44b4b885c8051c798e2a7, entries=150, sequenceid=283, filesize=11.9 K 2023-03-30 16:55:48,862 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 4038776cda6db2eaa29896eab9262a79 in 1258ms, sequenceid=283, compaction requested=true 2023-03-30 16:55:48,862 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:48,862 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:48,862 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:48,862 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:55:48,862 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:55:48,862 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:48,864 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48934 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:55:48,864 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/A is initiating minor compaction (all files) 2023-03-30 16:55:48,864 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/A in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:48,864 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/eb285f1e7b724a0dace42da0199d18c6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/5443d9ece19e4a138c2655229ec41bc8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/1a65f3e079d14bb9a46dab9a5fad2d82, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/e35878dea0e7430c8234c61dee9c8603] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=47.8 K 2023-03-30 16:55:48,865 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:55:48,865 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/B is initiating minor compaction (all files) 2023-03-30 16:55:48,865 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/B in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:48,865 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/48035ff5d48f4cf7b99f995e9b8106eb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/385a6f9105f648ffae0e4bff001a1b70, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/9fe1879d858149e6aede4670b393df75, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/3a7060e7ceba45a893c0933554c843a5] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=45.4 K 2023-03-30 16:55:48,865 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting eb285f1e7b724a0dace42da0199d18c6, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1680195345434 2023-03-30 16:55:48,866 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 5443d9ece19e4a138c2655229ec41bc8, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680195345791 2023-03-30 16:55:48,866 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 1a65f3e079d14bb9a46dab9a5fad2d82, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680195345930 2023-03-30 16:55:48,866 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting e35878dea0e7430c8234c61dee9c8603, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1680195346979 2023-03-30 16:55:48,866 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 48035ff5d48f4cf7b99f995e9b8106eb, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1680195345434 2023-03-30 16:55:48,867 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 385a6f9105f648ffae0e4bff001a1b70, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680195345791 2023-03-30 16:55:48,867 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 9fe1879d858149e6aede4670b393df75, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680195345930 2023-03-30 16:55:48,867 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 3a7060e7ceba45a893c0933554c843a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1680195346979 2023-03-30 16:55:48,873 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#A#compaction#401 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:48,877 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#B#compaction#402 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:48,883 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:48,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:48,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:48,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:48,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:48,893 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:48,893 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:48,893 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:48,893 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:48,893 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,893 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,894 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:48,894 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:48,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:48,895 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:48,895 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:48,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:48,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:48,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:48,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:48,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:48,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:48,897 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:48,897 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:48,897 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:48,897 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:48,897 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:48,897 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:48,897 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:48,897 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-30 16:55:48,897 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:48,897 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:48,898 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:48,898 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:48,898 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:48,898 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:48,899 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/f89e7a7348134ae481388deeeda727f8 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/f89e7a7348134ae481388deeeda727f8 2023-03-30 16:55:48,900 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/010bfe1f53d54c4787bac67092a8430d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/010bfe1f53d54c4787bac67092a8430d 2023-03-30 16:55:48,904 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/B of 4038776cda6db2eaa29896eab9262a79 into 010bfe1f53d54c4787bac67092a8430d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:48,904 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:48,904 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/B, priority=12, startTime=1680195348862; duration=0sec 2023-03-30 16:55:48,904 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:48,904 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:55:48,904 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/A of 4038776cda6db2eaa29896eab9262a79 into f89e7a7348134ae481388deeeda727f8(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:48,904 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:48,904 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/A, priority=12, startTime=1680195348862; duration=0sec 2023-03-30 16:55:48,905 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:48,905 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:55:48,905 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/C is initiating minor compaction (all files) 2023-03-30 16:55:48,905 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/C in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:48,905 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/9c18d9fc1b644437b10692954e6ad3a6 2023-03-30 16:55:48,905 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/3d51ffd9dc9c41e8afb92f8545a000be, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ae1dcb6aa9024402928a1849b73f2317, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/0f83127160924439bf06047b5ad8f033, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/f11f183a10e44b4b885c8051c798e2a7] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=45.4 K 2023-03-30 16:55:48,906 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 3d51ffd9dc9c41e8afb92f8545a000be, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1680195345434 2023-03-30 16:55:48,906 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting ae1dcb6aa9024402928a1849b73f2317, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1680195345791 2023-03-30 16:55:48,908 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 0f83127160924439bf06047b5ad8f033, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680195345930 2023-03-30 16:55:48,908 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting f11f183a10e44b4b885c8051c798e2a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1680195346979 2023-03-30 16:55:48,914 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#C#compaction#405 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:48,929 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/5256c400bd664fddaadc7538665f2333 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/5256c400bd664fddaadc7538665f2333 2023-03-30 16:55:48,934 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/C of 4038776cda6db2eaa29896eab9262a79 into 5256c400bd664fddaadc7538665f2333(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:48,934 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:48,934 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/C, priority=12, startTime=1680195348862; duration=0sec 2023-03-30 16:55:48,934 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:49,323 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/4f0ff42414e54766a02e13decaf87f03 2023-03-30 16:55:49,733 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/dc6000db51a54e28a4bed8e4018f3f22 2023-03-30 16:55:49,738 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/9c18d9fc1b644437b10692954e6ad3a6 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/9c18d9fc1b644437b10692954e6ad3a6 2023-03-30 16:55:49,743 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/9c18d9fc1b644437b10692954e6ad3a6, entries=150, sequenceid=302, filesize=11.9 K 2023-03-30 16:55:49,745 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/4f0ff42414e54766a02e13decaf87f03 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/4f0ff42414e54766a02e13decaf87f03 2023-03-30 16:55:49,755 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/4f0ff42414e54766a02e13decaf87f03, entries=150, sequenceid=302, filesize=11.9 K 2023-03-30 16:55:49,756 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/dc6000db51a54e28a4bed8e4018f3f22 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/dc6000db51a54e28a4bed8e4018f3f22 2023-03-30 16:55:49,761 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/dc6000db51a54e28a4bed8e4018f3f22, entries=150, sequenceid=302, filesize=11.9 K 2023-03-30 16:55:49,762 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 4038776cda6db2eaa29896eab9262a79 in 865ms, sequenceid=302, compaction requested=false 2023-03-30 16:55:49,762 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:49,762 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:49,762 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:49,762 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:49,762 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,763 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:49,763 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:49,765 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:49,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:49,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61b25eb6[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:49,767 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:49,767 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:49,774 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:49,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:49,774 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:49,774 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:49,774 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:49,774 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,774 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:49,776 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:49,776 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:49,776 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,776 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:49,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:49,783 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:49,783 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:49,783 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:49,783 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:49,783 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:49,785 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:49,785 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:49,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:49,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:49,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:49,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:49,786 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:49,788 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,789 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,796 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,796 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:49,796 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:49,796 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:49,796 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:49,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:49,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:49,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:49,796 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:49,796 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:49,796 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:49,796 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:49,796 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:49,796 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:49,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:49,797 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:49,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:49,802 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:55:49,803 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:49,803 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:49,803 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:49,803 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:49,803 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:49,803 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:49,820 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/e1e0da1d84ae42a1aaa31bc87ef3715c 2023-03-30 16:55:49,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1680195349803, max timestamp=1680195349830Num uniques 859; 2023-03-30 16:55:49,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-30 16:55:49,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1680195349803, max timestamp=1680195349830Num uniques 859; 2023-03-30 16:55:49,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-30 16:55:49,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1680195349803, max timestamp=1680195349830Num uniques 859; 2023-03-30 16:55:49,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-30 16:55:49,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:49,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:49,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195409845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195409846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:49,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195409846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:49,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195409852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,897 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:49,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:49,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:49,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:49,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:49,899 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:49,899 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:49,899 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:49,899 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:49,900 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:49,900 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:49,901 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:49,901 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:49,901 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,901 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:49,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:49,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:49,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:49,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:49,903 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:49,903 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:49,903 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:49,903 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:49,903 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:49,903 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:49,903 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:49,903 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:49,903 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:49,903 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:49,903 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:49,903 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,903 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:49,903 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:49,904 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,904 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:49,905 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:49,905 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:49,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:49,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b93b9e9[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:49,906 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:49,906 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:49,907 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:49,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:49,908 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:49,908 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:49,908 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,908 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:49,908 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:49,910 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,910 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:49,910 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:49,910 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:49,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:49,913 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:49,913 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:49,913 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:49,913 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:49,913 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:49,915 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:49,915 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:49,915 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:49,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:49,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:49,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:49,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:49,916 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:49,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:49,917 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,919 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,919 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:49,919 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:49,919 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,919 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:49,919 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:49,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:49,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:49,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:49,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:49,919 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:49,919 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:49,920 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:49,920 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:49,920 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:49,920 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:49,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:49,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:49,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:49,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195409953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:49,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195409954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:49,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57278 deadline: 1680195409954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:49,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:49,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195409954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,020 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:50,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:50,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:50,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:50,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:50,023 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:50,023 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:50,023 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:50,023 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:50,023 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,023 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:50,024 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:50,025 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,025 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,026 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:50,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:50,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:50,026 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:50,026 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:50,026 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:50,026 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:50,026 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:50,026 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:50,026 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:50,027 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:50,027 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:50,029 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,029 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:50,029 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,029 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:50,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:50,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f107bc7[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:50,031 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:50,031 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,032 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,032 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,032 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:50,032 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:50,032 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:50,032 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:50,034 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:50,034 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:50,034 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:50,034 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:50,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:50,037 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:50,037 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:50,037 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:50,037 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:50,037 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:50,038 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,038 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:50,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,038 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:50,039 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,041 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,044 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:50,044 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,044 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:50,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:50,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:50,044 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:50,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:50,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:50,044 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,045 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,045 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,045 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,045 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,045 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:50,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:50,045 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:50,045 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:50,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:50,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57264 deadline: 1680195410087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,093 DEBUG [Thread-2813] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:55:50,145 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:50,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:50,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:50,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:50,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:50,148 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:50,148 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:50,148 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:50,148 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:50,149 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,149 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,150 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:50,150 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:50,150 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,150 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:50,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:50,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:50,151 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:50,151 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:50,151 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:50,152 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:50,152 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:50,152 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:50,152 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:50,153 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,153 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:50,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:50,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b058c7f[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:50,155 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:50,155 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,156 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,156 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:50,156 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:50,156 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,156 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:50,156 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:50,159 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,159 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:50,159 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:50,159 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:50,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:50,162 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:50,162 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:50,162 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:50,162 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:50,162 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:50,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:50,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57280 deadline: 1680195410159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:50,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195410159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:50,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195410159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:50,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195410159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,164 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,164 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,164 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:50,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:50,164 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:50,166 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,171 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,171 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,171 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:50,171 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,171 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,171 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,171 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:50,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:50,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:50,171 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:50,171 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,171 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:50,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:50,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:50,172 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:50,173 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:50,229 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/e974090b7b6b47f492548b9309adeebd 2023-03-30 16:55:50,273 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:50,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:50,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:50,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:50,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:50,276 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:50,276 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:50,276 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:50,276 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:50,276 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,276 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,277 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:50,277 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:50,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:50,278 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,278 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:50,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:50,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:50,279 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:50,279 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:50,279 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:50,279 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:50,279 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:50,279 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:50,279 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:50,279 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:50,279 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:50,279 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:50,279 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:50,279 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:50,279 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:50,279 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,279 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:50,281 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:50,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:50,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48b13247[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:50,283 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:50,283 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,285 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:50,285 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,285 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:50,287 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:50,287 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,287 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:50,287 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:50,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:50,289 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:50,289 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:50,289 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:50,289 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:50,289 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:50,291 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,291 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:50,291 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:50,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:50,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:50,291 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,293 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,295 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,295 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:50,295 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,295 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:50,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:50,295 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:50,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:50,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:50,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:50,295 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,295 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:50,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:50,295 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,295 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:50,296 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:50,296 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,296 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,396 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:50,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:50,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:50,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:50,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:50,399 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:50,399 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:50,399 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:50,399 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:50,399 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,399 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,401 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:50,401 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:50,402 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,402 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:50,403 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:50,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:50,403 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:50,403 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:50,403 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:50,403 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:50,403 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:50,403 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:50,403 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:50,403 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:50,403 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:50,403 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:50,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:50,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:50,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:50,405 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,405 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:50,405 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,405 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:50,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:50,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e3b9649[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:50,407 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:50,407 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,408 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:50,408 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,408 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:50,409 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:50,409 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:50,409 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:50,409 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:50,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:50,412 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:50,412 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:50,412 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:50,412 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:50,412 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:50,414 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,414 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,414 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:50,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:50,415 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,416 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:50,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,418 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:50,418 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:50,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:50,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:50,418 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,418 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:50,418 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:50,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:50,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:50,418 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:50,418 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,418 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:50,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:50,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:50,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195410464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:50,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57252 deadline: 1680195410464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:50,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195410465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:50,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195410465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,519 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:50,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:50,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:50,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:50,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:50,522 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:50,522 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:50,522 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:50,522 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:50,522 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,522 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,523 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,523 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:50,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:50,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:50,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:50,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:50,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:50,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:50,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:50,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:50,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:50,525 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:50,525 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:50,525 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:50,525 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:50,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:50,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:50,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:50,525 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:50,527 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:50,527 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:50,527 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,527 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:50,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:50,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42a9080a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:50,528 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:50,528 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,530 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,530 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:50,530 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:50,530 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,530 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:50,530 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:50,532 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:50,532 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:50,532 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:50,532 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:50,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:50,535 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:50,535 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:50,535 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:50,535 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:50,535 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:50,537 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,538 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:50,538 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:50,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:50,538 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,540 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,543 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,543 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,543 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,543 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,543 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:50,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:50,543 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:50,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:50,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:50,543 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:50,543 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:50,543 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:50,543 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:50,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:50,543 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,543 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:50,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:50,638 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/167b390ce3cc44ecbe33419c1e74b4dd 2023-03-30 16:55:50,642 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/e1e0da1d84ae42a1aaa31bc87ef3715c as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/e1e0da1d84ae42a1aaa31bc87ef3715c 2023-03-30 16:55:50,643 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:50,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:50,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:50,646 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/e1e0da1d84ae42a1aaa31bc87ef3715c, entries=150, sequenceid=316, filesize=11.9 K 2023-03-30 16:55:50,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:50,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:50,646 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:50,646 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:50,646 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:50,646 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:50,646 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,647 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,647 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/e974090b7b6b47f492548b9309adeebd as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/e974090b7b6b47f492548b9309adeebd 2023-03-30 16:55:50,648 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,648 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:50,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:50,648 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,648 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:50,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:50,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:50,650 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:50,650 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:50,650 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:50,650 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:50,650 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:50,650 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:50,650 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:50,650 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:50,650 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:50,650 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:50,650 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:50,650 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,650 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:50,650 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:50,651 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/e974090b7b6b47f492548b9309adeebd, entries=100, sequenceid=316, filesize=9.5 K 2023-03-30 16:55:50,651 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/167b390ce3cc44ecbe33419c1e74b4dd as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/167b390ce3cc44ecbe33419c1e74b4dd 2023-03-30 16:55:50,654 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/167b390ce3cc44ecbe33419c1e74b4dd, entries=100, sequenceid=316, filesize=9.5 K 2023-03-30 16:55:50,655 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 4038776cda6db2eaa29896eab9262a79 in 853ms, sequenceid=316, compaction requested=true 2023-03-30 16:55:50,655 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:50,655 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:50,656 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:55:50,656 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:55:50,656 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:50,656 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:50,656 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,656 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:50,656 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,656 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:50,657 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37134 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:55:50,657 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34694 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:55:50,657 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/A is initiating minor compaction (all files) 2023-03-30 16:55:50,657 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/B is initiating minor compaction (all files) 2023-03-30 16:55:50,657 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/A in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:50,657 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/B in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:50,657 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/010bfe1f53d54c4787bac67092a8430d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/4f0ff42414e54766a02e13decaf87f03, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/e974090b7b6b47f492548b9309adeebd] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=33.9 K 2023-03-30 16:55:50,657 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/f89e7a7348134ae481388deeeda727f8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/9c18d9fc1b644437b10692954e6ad3a6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/e1e0da1d84ae42a1aaa31bc87ef3715c] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=36.3 K 2023-03-30 16:55:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,657 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 010bfe1f53d54c4787bac67092a8430d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1680195346979 2023-03-30 16:55:50,657 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting f89e7a7348134ae481388deeeda727f8, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1680195346979 2023-03-30 16:55:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,658 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 4f0ff42414e54766a02e13decaf87f03, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680195347627 2023-03-30 16:55:50,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:50,658 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 9c18d9fc1b644437b10692954e6ad3a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680195347627 2023-03-30 16:55:50,658 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting e974090b7b6b47f492548b9309adeebd, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1680195349791 2023-03-30 16:55:50,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:50,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55580254[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:50,658 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:50,658 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting e1e0da1d84ae42a1aaa31bc87ef3715c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1680195349791 2023-03-30 16:55:50,658 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,662 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,662 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,662 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:50,662 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:50,662 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:50,662 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:50,664 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:50,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:50,664 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:50,664 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:50,664 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:50,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:50,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:50,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:50,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:50,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:50,666 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:50,666 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:50,666 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:50,666 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:50,666 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:50,667 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:50,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:50,668 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,668 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:50,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:50,668 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:50,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:50,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:50,669 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#A#compaction#410 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:50,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:50,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,671 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#B#compaction#411 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:50,672 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,676 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,676 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:50,676 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:50,677 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:50,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:50,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:50,676 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:50,677 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:50,677 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:50,676 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:50,677 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,677 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,677 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:50,677 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:50,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:50,679 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/4bab7b5724d646c38aa3c985607bc124 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/4bab7b5724d646c38aa3c985607bc124 2023-03-30 16:55:50,682 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/B of 4038776cda6db2eaa29896eab9262a79 into 4bab7b5724d646c38aa3c985607bc124(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:50,682 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:50,682 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/B, priority=13, startTime=1680195350656; duration=0sec 2023-03-30 16:55:50,682 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:50,682 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:55:50,685 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34694 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:55:50,685 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/C is initiating minor compaction (all files) 2023-03-30 16:55:50,685 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/C in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:50,685 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/5256c400bd664fddaadc7538665f2333, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/dc6000db51a54e28a4bed8e4018f3f22, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/167b390ce3cc44ecbe33419c1e74b4dd] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=33.9 K 2023-03-30 16:55:50,685 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 5256c400bd664fddaadc7538665f2333, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1680195346979 2023-03-30 16:55:50,685 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting dc6000db51a54e28a4bed8e4018f3f22, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680195347627 2023-03-30 16:55:50,686 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 167b390ce3cc44ecbe33419c1e74b4dd, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1680195349791 2023-03-30 16:55:50,690 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#C#compaction#412 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:50,703 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/12c95776af3945dc96467cc26c24f5bc as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/12c95776af3945dc96467cc26c24f5bc 2023-03-30 16:55:50,706 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/C of 4038776cda6db2eaa29896eab9262a79 into 12c95776af3945dc96467cc26c24f5bc(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:50,706 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:50,706 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/C, priority=13, startTime=1680195350656; duration=0sec 2023-03-30 16:55:50,707 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:50,777 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:50,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:50,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:50,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:50,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:50,787 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:50,787 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:50,787 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:50,787 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:50,788 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,788 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,790 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:50,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,790 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:50,791 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,791 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:50,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:50,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:50,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:50,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:50,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:50,792 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:50,792 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:50,792 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:50,796 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:50,796 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:50,796 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:50,796 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:50,796 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=134.18 KB heapSize=361.55 KB 2023-03-30 16:55:50,796 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:50,796 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:50,796 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:50,796 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:50,796 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:50,796 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:50,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:50,975 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:50,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:50,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195410983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:50,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195410987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:50,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195410988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:50,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:50,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57194 deadline: 1680195410991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,077 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/5b181c2e0a864813b088aa4a000735ae as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/5b181c2e0a864813b088aa4a000735ae 2023-03-30 16:55:51,081 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/A of 4038776cda6db2eaa29896eab9262a79 into 5b181c2e0a864813b088aa4a000735ae(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:51,081 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:51,081 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/A, priority=13, startTime=1680195350655; duration=0sec 2023-03-30 16:55:51,081 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:51,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:51,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195411092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:51,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195411092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:51,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195411096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:51,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195411098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,218 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/70c5653ac0a1488a87ebc147154109d1 2023-03-30 16:55:51,240 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/83d3915746b64cea84dd9fc8950a2e47 2023-03-30 16:55:51,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:51,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195411299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:51,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195411299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:51,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195411299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:51,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195411300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:51,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195411604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:51,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195411605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:51,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195411606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:51,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195411606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,662 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/4f9a6ae12f934fb881b0044ff9143f31 2023-03-30 16:55:51,667 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/70c5653ac0a1488a87ebc147154109d1 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/70c5653ac0a1488a87ebc147154109d1 2023-03-30 16:55:51,671 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/70c5653ac0a1488a87ebc147154109d1, entries=150, sequenceid=350, filesize=11.9 K 2023-03-30 16:55:51,672 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/83d3915746b64cea84dd9fc8950a2e47 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/83d3915746b64cea84dd9fc8950a2e47 2023-03-30 16:55:51,704 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/83d3915746b64cea84dd9fc8950a2e47, entries=150, sequenceid=350, filesize=11.9 K 2023-03-30 16:55:51,705 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/4f9a6ae12f934fb881b0044ff9143f31 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/4f9a6ae12f934fb881b0044ff9143f31 2023-03-30 16:55:51,709 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/4f9a6ae12f934fb881b0044ff9143f31, entries=150, sequenceid=350, filesize=11.9 K 2023-03-30 16:55:51,710 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~361.50 KB/370176, currentSize=53.67 KB/54960 for 4038776cda6db2eaa29896eab9262a79 in 914ms, sequenceid=350, compaction requested=false 2023-03-30 16:55:51,710 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:51,710 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:51,710 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:51,710 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:51,710 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,710 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:51,710 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:51,712 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:51,712 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:51,713 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:51,713 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:51,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:51,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14ba21b2[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:51,717 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:51,717 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:51,719 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:51,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:51,719 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:51,719 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:51,719 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:51,719 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,719 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:51,721 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:51,721 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:51,721 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:51,722 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:51,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:51,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:51,725 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:51,725 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:51,725 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:51,725 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:51,725 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:51,727 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:51,727 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:51,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:51,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:51,728 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:51,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:51,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:51,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:51,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:51,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:51,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:51,744 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:51,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:51,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:51,748 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:51,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,755 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:51,755 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:51,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:51,755 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:51,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:51,756 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:51,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:51,756 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:51,756 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:51,756 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,756 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:51,756 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:51,756 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:51,756 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:51,756 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:51,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:51,856 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:51,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:51,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:51,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:51,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:51,860 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:51,860 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:51,860 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:51,860 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:51,861 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:51,861 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:51,864 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,864 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:51,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:51,864 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:51,864 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:51,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:51,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:51,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:51,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:51,866 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:51,866 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:51,866 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 16:55:51,866 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:51,866 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:51,866 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:51,867 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:51,867 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:51,867 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:52,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:52,118 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:52,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680195352134, max timestamp=1680195352144Num uniques 859; 2023-03-30 16:55:52,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 16:55:52,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680195352134, max timestamp=1680195352144Num uniques 859; 2023-03-30 16:55:52,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680195352134, max timestamp=1680195352144Num uniques 859; 2023-03-30 16:55:52,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 16:55:52,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 16:55:52,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:52,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195412169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:52,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:52,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195412170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:52,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:52,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195412171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:52,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:52,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195412172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:52,278 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/b8a6f112dc8c4c9ebe19fc11a424702a 2023-03-30 16:55:52,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:52,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195412273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:52,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:52,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195412278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:52,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:52,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195412278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:52,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:52,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195412280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:52,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:52,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195412481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:52,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:52,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195412484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:52,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:52,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195412485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:52,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:52,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195412486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:52,688 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/ac2a5302097d4a82badad9d394f79f1a 2023-03-30 16:55:52,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:52,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195412787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:52,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:52,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195412791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:52,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:52,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195412793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:52,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:52,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195412794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,099 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/0052b6e09d53407495a2f8ae0ef96896 2023-03-30 16:55:53,104 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/b8a6f112dc8c4c9ebe19fc11a424702a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/b8a6f112dc8c4c9ebe19fc11a424702a 2023-03-30 16:55:53,108 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/b8a6f112dc8c4c9ebe19fc11a424702a, entries=150, sequenceid=362, filesize=11.9 K 2023-03-30 16:55:53,108 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/ac2a5302097d4a82badad9d394f79f1a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/ac2a5302097d4a82badad9d394f79f1a 2023-03-30 16:55:53,111 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/ac2a5302097d4a82badad9d394f79f1a, entries=150, sequenceid=362, filesize=11.9 K 2023-03-30 16:55:53,112 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/0052b6e09d53407495a2f8ae0ef96896 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/0052b6e09d53407495a2f8ae0ef96896 2023-03-30 16:55:53,115 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/0052b6e09d53407495a2f8ae0ef96896, entries=150, sequenceid=362, filesize=11.9 K 2023-03-30 16:55:53,115 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 4038776cda6db2eaa29896eab9262a79 in 1249ms, sequenceid=362, compaction requested=true 2023-03-30 16:55:53,115 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:53,115 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:53,115 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:53,115 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:53,115 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,115 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:53,115 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:53,118 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:53,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:53,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:53,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:53,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:53,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@567164b2[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:53,120 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:53,120 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,122 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,122 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:53,122 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:53,122 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,122 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:53,122 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:53,123 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:53,123 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,123 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:53,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:53,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:53,123 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:53,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:53,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:53,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:53,126 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:53,126 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:53,126 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:53,126 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:53,126 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:53,133 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,133 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:53,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,133 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:53,134 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:53,135 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:53,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:53,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,138 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,138 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:53,138 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:53,138 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:53,138 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:53,138 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:53,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:53,138 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:53,138 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:53,138 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:53,138 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:53,138 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,138 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:53,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:53,238 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:53,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:53,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:53,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:53,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:53,241 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:53,241 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:53,241 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:53,241 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:53,242 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,242 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:53,243 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:53,243 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:53,243 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,243 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:53,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:53,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:53,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:53,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:53,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:53,245 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:53,245 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:53,245 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:53,245 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:53,245 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:53,245 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:53,245 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:53,245 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-30 16:55:53,245 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:53,245 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:53,245 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:53,245 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:53,245 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:53,246 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:53,252 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/e8f27cfe61f14baea1508ace7908a9b3 2023-03-30 16:55:53,263 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/0cae7d54490441379e7ec8837e55e25c 2023-03-30 16:55:53,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:53,303 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:53,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:53,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195413313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:53,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195413315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:53,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195413316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:53,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195413317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:53,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195413423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:53,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195413424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:53,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195413424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:53,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195413425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:53,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195413628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:53,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195413628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:53,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195413628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:53,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195413630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,682 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/979fe09249bc407790077e0b7f44659e 2023-03-30 16:55:53,686 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/e8f27cfe61f14baea1508ace7908a9b3 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/e8f27cfe61f14baea1508ace7908a9b3 2023-03-30 16:55:53,689 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/e8f27cfe61f14baea1508ace7908a9b3, entries=150, sequenceid=396, filesize=11.9 K 2023-03-30 16:55:53,690 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/0cae7d54490441379e7ec8837e55e25c as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/0cae7d54490441379e7ec8837e55e25c 2023-03-30 16:55:53,693 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/0cae7d54490441379e7ec8837e55e25c, entries=150, sequenceid=396, filesize=11.9 K 2023-03-30 16:55:53,694 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/979fe09249bc407790077e0b7f44659e as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/979fe09249bc407790077e0b7f44659e 2023-03-30 16:55:53,697 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/979fe09249bc407790077e0b7f44659e, entries=150, sequenceid=396, filesize=11.9 K 2023-03-30 16:55:53,697 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 4038776cda6db2eaa29896eab9262a79 in 452ms, sequenceid=396, compaction requested=true 2023-03-30 16:55:53,697 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:53,698 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:53,698 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:53,698 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:53,698 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,698 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:53,698 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:53,700 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,700 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:53,700 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,700 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:53,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:53,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c12cf2e[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:53,702 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:53,702 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,704 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,704 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:53,704 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:53,704 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:53,704 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,704 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:53,705 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:53,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:53,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:53,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:53,708 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:53,708 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:53,708 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:53,708 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:53,708 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:53,709 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,709 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,709 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:53,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:53,710 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:53,711 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:53,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:53,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:53,713 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:53,713 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,713 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:53,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:53,713 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:53,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:53,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:53,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:53,713 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:53,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:53,714 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:53,714 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:53,714 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:53,714 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,714 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,714 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,714 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,814 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:53,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:53,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:53,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:53,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:53,820 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:53,820 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:53,820 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:53,820 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:53,820 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,820 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:53,823 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:53,823 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:53,823 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,823 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:53,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:53,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:53,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:53,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:53,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:53,829 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:53,829 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:53,829 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:53,832 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:53,832 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:53,832 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:53,832 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:53,833 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 16:55:53,833 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:53,833 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:53,833 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:53,833 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:53,833 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:53,833 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:53,846 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/69d0bd42070843b2a8c19a115fe7806b 2023-03-30 16:55:53,867 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/7c4941338a4f46e4813d6192148fe38f 2023-03-30 16:55:53,889 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/57aaf2ca7f174cc0b46b850a30bcf5dc 2023-03-30 16:55:53,893 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/69d0bd42070843b2a8c19a115fe7806b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/69d0bd42070843b2a8c19a115fe7806b 2023-03-30 16:55:53,896 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/69d0bd42070843b2a8c19a115fe7806b, entries=100, sequenceid=407, filesize=9.5 K 2023-03-30 16:55:53,896 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/7c4941338a4f46e4813d6192148fe38f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/7c4941338a4f46e4813d6192148fe38f 2023-03-30 16:55:53,899 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/7c4941338a4f46e4813d6192148fe38f, entries=100, sequenceid=407, filesize=9.5 K 2023-03-30 16:55:53,899 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/57aaf2ca7f174cc0b46b850a30bcf5dc as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/57aaf2ca7f174cc0b46b850a30bcf5dc 2023-03-30 16:55:53,903 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/57aaf2ca7f174cc0b46b850a30bcf5dc, entries=100, sequenceid=407, filesize=9.5 K 2023-03-30 16:55:53,903 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 4038776cda6db2eaa29896eab9262a79 in 70ms, sequenceid=407, compaction requested=true 2023-03-30 16:55:53,903 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:53,903 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:53,903 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:53,903 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:53,903 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,903 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:53,903 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:53,905 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,905 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:53,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:53,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:53,906 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,906 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:53,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:53,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:53,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:53,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49cd2c5a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:53,907 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:53,907 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,908 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,909 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:53,909 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:53,909 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:53,909 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,909 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:53,910 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,910 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:53,910 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:53,910 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:53,913 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:53,913 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:53,913 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:53,913 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:53,913 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:53,920 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,920 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:53,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,920 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:53,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:53,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:53,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:53,920 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,923 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,925 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,925 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:53,925 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:53,925 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:53,925 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,925 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:53,925 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:53,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:53,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:53,925 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:53,925 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:53,925 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:53,925 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:53,925 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:53,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:53,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:53,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:53,944 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:55:53,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:53,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:53,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:53,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:53,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:53,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:53,951 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=417 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/4e91b2997b934d92947b95dc61597f9a 2023-03-30 16:55:53,959 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=417 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/437757916c014c8eaf377a8b69feb596 2023-03-30 16:55:53,975 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=417 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/46b588791c95473ba06dcf19e070e8b7 2023-03-30 16:55:53,979 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/4e91b2997b934d92947b95dc61597f9a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/4e91b2997b934d92947b95dc61597f9a 2023-03-30 16:55:53,982 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/4e91b2997b934d92947b95dc61597f9a, entries=250, sequenceid=417, filesize=16.7 K 2023-03-30 16:55:53,982 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/437757916c014c8eaf377a8b69feb596 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/437757916c014c8eaf377a8b69feb596 2023-03-30 16:55:53,985 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/437757916c014c8eaf377a8b69feb596, entries=150, sequenceid=417, filesize=11.9 K 2023-03-30 16:55:53,986 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/46b588791c95473ba06dcf19e070e8b7 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/46b588791c95473ba06dcf19e070e8b7 2023-03-30 16:55:53,989 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/46b588791c95473ba06dcf19e070e8b7, entries=150, sequenceid=417, filesize=11.9 K 2023-03-30 16:55:53,989 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=107.34 KB/109920 for 4038776cda6db2eaa29896eab9262a79 in 45ms, sequenceid=417, compaction requested=true 2023-03-30 16:55:53,990 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:53,990 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:53,990 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:53,990 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-30 16:55:53,990 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:53,990 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-30 16:55:53,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:53,991 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-30 16:55:53,991 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:53,992 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:53,992 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:53,992 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:53,992 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:53,992 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:53,992 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71354 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-30 16:55:53,992 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/B is initiating minor compaction (all files) 2023-03-30 16:55:53,992 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76234 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-30 16:55:53,992 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/B in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:53,992 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/A is initiating minor compaction (all files) 2023-03-30 16:55:53,992 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/4bab7b5724d646c38aa3c985607bc124, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/83d3915746b64cea84dd9fc8950a2e47, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/ac2a5302097d4a82badad9d394f79f1a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/0cae7d54490441379e7ec8837e55e25c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/7c4941338a4f46e4813d6192148fe38f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/437757916c014c8eaf377a8b69feb596] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=69.7 K 2023-03-30 16:55:53,992 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/A in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:53,993 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/5b181c2e0a864813b088aa4a000735ae, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/70c5653ac0a1488a87ebc147154109d1, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/b8a6f112dc8c4c9ebe19fc11a424702a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/e8f27cfe61f14baea1508ace7908a9b3, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/69d0bd42070843b2a8c19a115fe7806b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/4e91b2997b934d92947b95dc61597f9a] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=74.4 K 2023-03-30 16:55:53,993 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 4bab7b5724d646c38aa3c985607bc124, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1680195347634 2023-03-30 16:55:53,993 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 5b181c2e0a864813b088aa4a000735ae, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1680195347634 2023-03-30 16:55:53,993 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 83d3915746b64cea84dd9fc8950a2e47, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1680195349837 2023-03-30 16:55:53,993 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 70c5653ac0a1488a87ebc147154109d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1680195349837 2023-03-30 16:55:53,994 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting ac2a5302097d4a82badad9d394f79f1a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680195350981 2023-03-30 16:55:53,994 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting b8a6f112dc8c4c9ebe19fc11a424702a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680195350981 2023-03-30 16:55:53,994 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 0cae7d54490441379e7ec8837e55e25c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1680195352168 2023-03-30 16:55:53,994 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting e8f27cfe61f14baea1508ace7908a9b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1680195352168 2023-03-30 16:55:53,994 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 7c4941338a4f46e4813d6192148fe38f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680195353312 2023-03-30 16:55:53,994 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 69d0bd42070843b2a8c19a115fe7806b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680195353312 2023-03-30 16:55:53,995 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 437757916c014c8eaf377a8b69feb596, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=417, earliestPutTs=1680195353936 2023-03-30 16:55:53,995 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 4e91b2997b934d92947b95dc61597f9a, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=417, earliestPutTs=1680195353936 2023-03-30 16:55:54,006 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#A#compaction#430 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:54,006 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#B#compaction#429 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:54,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195414006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195414008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195414009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195414012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,025 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:54,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:54,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:54,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:54,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:54,028 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:54,028 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:54,028 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:54,028 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:54,029 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,029 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,030 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,030 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:54,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:54,030 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,030 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,031 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:54,031 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:54,031 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:54,031 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:54,031 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:54,031 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:54,031 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:54,032 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:54,031 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:54,032 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:54,032 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:54,032 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:54,032 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:54,032 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:54,032 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,032 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:54,032 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:54,034 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,034 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:54,034 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,035 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:54,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:54,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c3f3360[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:54,041 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:54,041 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,043 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,043 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:54,043 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:54,043 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,043 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:54,043 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:54,044 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:54,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:54,044 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:54,044 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:54,044 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:54,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:54,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:54,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:54,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:54,046 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:54,046 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:54,046 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:54,046 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:54,046 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:54,048 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,048 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,048 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:54,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:54,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:54,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:54,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:54,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:54,049 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,050 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,052 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,052 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:54,052 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,052 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:54,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:54,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:54,052 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,052 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,052 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,052 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,052 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:54,052 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:54,052 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:54,053 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:54,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:54,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:54,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57264 deadline: 1680195414096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,103 DEBUG [Thread-2813] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:55:54,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195414116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195414116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195414118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195414118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,153 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:54,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:54,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:54,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:54,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:54,156 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:54,156 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:54,156 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:54,156 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:54,156 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,156 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,159 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,159 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:54,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:54,159 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,159 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:54,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:54,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:54,160 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:54,160 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:54,160 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:54,160 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:54,160 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:54,160 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:54,160 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:54,160 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:54,160 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:54,160 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:54,160 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:54,161 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,161 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:54,161 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:54,162 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:54,162 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:54,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:54,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@513682a1[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:54,164 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:54,164 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,165 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:54,165 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:54,165 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,165 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:54,166 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,166 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:54,166 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:54,166 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:54,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:54,169 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:54,169 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:54,169 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:54,169 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:54,169 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:54,174 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,174 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:54,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,174 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:54,175 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,176 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,178 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:54,178 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:54,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:54,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:54,178 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:54,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:54,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:54,178 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:54,178 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:54,178 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:54,178 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,179 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,179 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,179 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,179 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,179 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:54,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:54,279 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:54,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:54,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:54,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:54,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:54,282 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:54,282 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:54,282 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:54,282 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:54,282 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,282 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,283 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:54,283 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:54,284 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,284 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:54,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:54,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:54,285 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:54,285 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:54,285 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:54,285 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:54,285 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:54,285 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:54,285 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:54,285 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:54,286 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:54,286 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:54,286 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:54,286 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,286 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:54,286 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:54,287 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,287 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:54,288 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,288 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b974c16[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:54,289 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:54,289 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,291 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,291 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:54,291 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:54,291 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:54,292 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,292 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:54,293 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:54,293 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:54,293 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,293 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:54,295 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:54,295 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:54,295 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:54,295 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:54,295 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:54,297 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,297 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:54,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,297 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:54,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:54,297 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:54,298 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:54,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,301 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:54,301 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,301 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:54,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:54,301 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:54,301 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,301 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,301 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,301 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,301 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,301 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:54,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:54,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:54,302 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:54,302 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:54,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:54,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195414323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195414323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195414324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195414324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,402 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:54,402 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/f40907c483e94b46a1f7b56527f3c200 2023-03-30 16:55:54,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:54,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:54,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:54,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:54,405 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:54,405 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:54,405 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:54,405 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:54,406 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,406 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,407 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:54,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:54,408 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,409 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,409 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:54,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:54,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:54,411 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:54,411 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:54,411 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:54,411 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:54,411 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:54,411 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:54,411 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:54,411 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:54,411 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:54,411 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:54,413 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/11a48877bd784c10a91bc7079d99e3be 2023-03-30 16:55:54,413 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,413 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:54,414 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,414 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,415 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/74092bd1cbf64ca1ac9bba341b572386 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/74092bd1cbf64ca1ac9bba341b572386 2023-03-30 16:55:54,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:54,421 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/514e98c2d7544119b001bf914a5cb63f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/514e98c2d7544119b001bf914a5cb63f 2023-03-30 16:55:54,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:54,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b981cc2[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:54,421 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:54,421 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,422 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,422 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:54,422 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:54,422 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:54,423 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:54,423 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,424 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:54,424 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:54,424 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:54,424 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:54,424 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:54,424 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:54,424 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:54,424 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:54,424 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:54,426 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/A of 4038776cda6db2eaa29896eab9262a79 into 74092bd1cbf64ca1ac9bba341b572386(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:54,426 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:54,426 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/A, priority=10, startTime=1680195353990; duration=0sec 2023-03-30 16:55:54,426 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:54,427 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-30 16:55:54,427 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,427 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:54,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,427 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:54,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:54,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:54,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:54,428 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/B of 4038776cda6db2eaa29896eab9262a79 into 514e98c2d7544119b001bf914a5cb63f(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:54,428 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:54,428 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/B, priority=10, startTime=1680195353990; duration=0sec 2023-03-30 16:55:54,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:54,428 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:54,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,428 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,429 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71354 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-30 16:55:54,429 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/C is initiating minor compaction (all files) 2023-03-30 16:55:54,429 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/C in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:54,429 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/12c95776af3945dc96467cc26c24f5bc, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/4f9a6ae12f934fb881b0044ff9143f31, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/0052b6e09d53407495a2f8ae0ef96896, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/979fe09249bc407790077e0b7f44659e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/57aaf2ca7f174cc0b46b850a30bcf5dc, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/46b588791c95473ba06dcf19e070e8b7] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=69.7 K 2023-03-30 16:55:54,430 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/5bab6431ab1e48debc03b845906b7685 2023-03-30 16:55:54,430 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 12c95776af3945dc96467cc26c24f5bc, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1680195347634 2023-03-30 16:55:54,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:54,430 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 4f9a6ae12f934fb881b0044ff9143f31, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1680195349837 2023-03-30 16:55:54,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,430 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,431 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 0052b6e09d53407495a2f8ae0ef96896, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680195350981 2023-03-30 16:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,431 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 979fe09249bc407790077e0b7f44659e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1680195352168 2023-03-30 16:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:54,431 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 57aaf2ca7f174cc0b46b850a30bcf5dc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680195353312 2023-03-30 16:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:54,432 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 46b588791c95473ba06dcf19e070e8b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=417, earliestPutTs=1680195353936 2023-03-30 16:55:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,433 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,433 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,433 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,433 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,433 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:54,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:54,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:54,433 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:54,433 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:54,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:54,433 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:54,433 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:54,433 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:54,433 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,433 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:54,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:54,434 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/f40907c483e94b46a1f7b56527f3c200 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/f40907c483e94b46a1f7b56527f3c200 2023-03-30 16:55:54,441 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/f40907c483e94b46a1f7b56527f3c200, entries=250, sequenceid=437, filesize=16.7 K 2023-03-30 16:55:54,442 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/11a48877bd784c10a91bc7079d99e3be as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/11a48877bd784c10a91bc7079d99e3be 2023-03-30 16:55:54,443 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#C#compaction#433 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:54,446 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/11a48877bd784c10a91bc7079d99e3be, entries=150, sequenceid=437, filesize=11.9 K 2023-03-30 16:55:54,446 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/5bab6431ab1e48debc03b845906b7685 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/5bab6431ab1e48debc03b845906b7685 2023-03-30 16:55:54,450 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/5bab6431ab1e48debc03b845906b7685, entries=150, sequenceid=437, filesize=11.9 K 2023-03-30 16:55:54,451 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 4038776cda6db2eaa29896eab9262a79 in 460ms, sequenceid=437, compaction requested=false 2023-03-30 16:55:54,451 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:54,533 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:54,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:54,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:54,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:54,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:54,536 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:54,536 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:54,536 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:54,536 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:54,537 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,537 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,538 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:54,538 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,538 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,539 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:54,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:54,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:54,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:54,539 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:54,539 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:54,540 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:54,540 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:54,540 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:54,540 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:54,540 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:54,540 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-30 16:55:54,540 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:54,540 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:54,540 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:54,540 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:54,540 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:54,540 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:54,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:54,638 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:54,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195414685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195414697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195414698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195414698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195414799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195414808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:54,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195414811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195414811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:54,854 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/bc18dafd25ee4bfe803ced03086f2ca9 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/bc18dafd25ee4bfe803ced03086f2ca9 2023-03-30 16:55:54,857 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/C of 4038776cda6db2eaa29896eab9262a79 into bc18dafd25ee4bfe803ced03086f2ca9(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:54,858 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:54,858 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/C, priority=10, startTime=1680195353990; duration=0sec 2023-03-30 16:55:54,858 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:54,949 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/239d3dfa88614696adebb1ab8c4a9d3f 2023-03-30 16:55:55,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:55,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195415004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:55,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195415012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:55,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195415019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:55,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195415019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:55,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195415307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:55,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195415315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:55,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195415322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:55,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195415324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,359 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/3e46f84b8dd646078c7ca9b72f5af59d 2023-03-30 16:55:55,368 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/a3cd8ce62b9f44f4a3989a5b8e2902d6 2023-03-30 16:55:55,372 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/239d3dfa88614696adebb1ab8c4a9d3f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/239d3dfa88614696adebb1ab8c4a9d3f 2023-03-30 16:55:55,376 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/239d3dfa88614696adebb1ab8c4a9d3f, entries=150, sequenceid=451, filesize=11.9 K 2023-03-30 16:55:55,376 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/3e46f84b8dd646078c7ca9b72f5af59d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/3e46f84b8dd646078c7ca9b72f5af59d 2023-03-30 16:55:55,386 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/3e46f84b8dd646078c7ca9b72f5af59d, entries=150, sequenceid=451, filesize=11.9 K 2023-03-30 16:55:55,387 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/a3cd8ce62b9f44f4a3989a5b8e2902d6 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/a3cd8ce62b9f44f4a3989a5b8e2902d6 2023-03-30 16:55:55,391 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/a3cd8ce62b9f44f4a3989a5b8e2902d6, entries=150, sequenceid=451, filesize=11.9 K 2023-03-30 16:55:55,391 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4038776cda6db2eaa29896eab9262a79 in 851ms, sequenceid=451, compaction requested=true 2023-03-30 16:55:55,391 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:55,392 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:55,392 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:55,393 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,393 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:55,394 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:55,394 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:55,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:55,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e499ce2[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:55,395 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:55,395 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:55,402 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:55,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:55,402 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:55,402 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,402 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:55,402 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:55,402 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:55,404 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:55,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:55,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:55,406 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:55,406 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:55,406 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:55,406 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:55,406 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:55,409 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:55,409 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:55,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:55,409 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:55,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:55,410 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,411 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:55,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:55,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:55,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:55,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:55,419 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:55,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:55,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:55,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:55,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:55,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:55,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:55,420 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:55,420 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:55,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:55,520 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:55,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:55,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:55,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:55,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:55,523 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:55,523 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:55,523 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:55,523 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:55,523 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:55,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:55,525 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:55,525 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:55,525 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,525 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:55,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:55,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:55,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:55,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:55,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:55,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:55,526 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:55,526 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:55,526 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:55,526 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:55,526 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:55,526 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:55,526 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:55,526 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-30 16:55:55,527 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:55,527 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:55,527 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:55,527 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:55,527 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:55,527 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:55,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:55,826 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:55,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:55,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195415834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:55,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195415836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:55,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195415843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:55,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195415844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,936 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=472 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/7233dbe2aee84c268a7738800b8d43ba 2023-03-30 16:55:55,945 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=472 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/56bbfbb8c3dd4f9ba36628ab9718e8a6 2023-03-30 16:55:55,951 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:55,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195415945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,952 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=472 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/4fe94ab84c1f47289dd0d4c5c1dce743 2023-03-30 16:55:55,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:55,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195415946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:55,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195415950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:55,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195415950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,968 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/7233dbe2aee84c268a7738800b8d43ba as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/7233dbe2aee84c268a7738800b8d43ba 2023-03-30 16:55:55,973 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/7233dbe2aee84c268a7738800b8d43ba, entries=150, sequenceid=472, filesize=11.9 K 2023-03-30 16:55:55,974 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/56bbfbb8c3dd4f9ba36628ab9718e8a6 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/56bbfbb8c3dd4f9ba36628ab9718e8a6 2023-03-30 16:55:55,978 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/56bbfbb8c3dd4f9ba36628ab9718e8a6, entries=150, sequenceid=472, filesize=11.9 K 2023-03-30 16:55:55,978 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/4fe94ab84c1f47289dd0d4c5c1dce743 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/4fe94ab84c1f47289dd0d4c5c1dce743 2023-03-30 16:55:55,983 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/4fe94ab84c1f47289dd0d4c5c1dce743, entries=150, sequenceid=472, filesize=11.9 K 2023-03-30 16:55:55,984 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 4038776cda6db2eaa29896eab9262a79 in 458ms, sequenceid=472, compaction requested=true 2023-03-30 16:55:55,984 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:55,984 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:55,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:55,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:55,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,987 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:55,987 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:55,992 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:55,992 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:55,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:55,993 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:55,993 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:55,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:55,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b0f2c00[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:55,995 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:55,995 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:55,996 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:55,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:55,996 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,996 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:55,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:55,999 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:55,999 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:56,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:56,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:56,003 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:56,003 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:56,003 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:56,003 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:56,003 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:56,005 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:56,005 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:56,005 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:56,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:56,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:56,007 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:56,010 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,013 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:56,013 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:56,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:56,013 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,013 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:56,013 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:56,013 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:56,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:56,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:56,013 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,013 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:56,013 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:56,013 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:56,014 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:56,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:56,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:56,014 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:56,114 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:56,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:56,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:56,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:56,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:56,117 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:56,117 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:56,117 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:56,117 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:56,118 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:56,118 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:56,119 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:56,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:56,119 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:56,119 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:56,120 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:56,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:56,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:56,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:56,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:56,121 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:56,121 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:56,121 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:56,122 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:56,122 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:56,122 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:56,122 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:56,122 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-30 16:55:56,123 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:56,123 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:56,123 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:56,123 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:56,123 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:56,123 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:56,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:56,157 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:56,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195356173, max timestamp=1680195356184Num uniques 859; 2023-03-30 16:55:56,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195356173, max timestamp=1680195356184Num uniques 859; 2023-03-30 16:55:56,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:55:56,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680195356173, max timestamp=1680195356184Num uniques 859; 2023-03-30 16:55:56,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:55:56,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:55:56,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:56,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195416218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:56,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195416223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:56,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195416224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:56,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195416224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:56,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195416329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:56,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195416334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:56,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195416334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:56,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195416335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,538 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/69a034d1bf174d6e983d6b7002fe1e2c 2023-03-30 16:55:56,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:56,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195416542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:56,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195416542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:56,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195416542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:56,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195416542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:56,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195416854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:56,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195416854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:56,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195416854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:56,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195416855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:56,955 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/2517ec3e65994aab9dd0f433e1ed3a7e 2023-03-30 16:55:57,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:57,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195417362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:57,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195417363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:57,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195417363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:57,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195417365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,399 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/6962f7188e1b425abde25c6cbb64efec 2023-03-30 16:55:57,403 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/69a034d1bf174d6e983d6b7002fe1e2c as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/69a034d1bf174d6e983d6b7002fe1e2c 2023-03-30 16:55:57,409 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/69a034d1bf174d6e983d6b7002fe1e2c, entries=100, sequenceid=484, filesize=9.5 K 2023-03-30 16:55:57,410 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/2517ec3e65994aab9dd0f433e1ed3a7e as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/2517ec3e65994aab9dd0f433e1ed3a7e 2023-03-30 16:55:57,413 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/2517ec3e65994aab9dd0f433e1ed3a7e, entries=100, sequenceid=484, filesize=9.5 K 2023-03-30 16:55:57,416 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/6962f7188e1b425abde25c6cbb64efec as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/6962f7188e1b425abde25c6cbb64efec 2023-03-30 16:55:57,426 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/6962f7188e1b425abde25c6cbb64efec, entries=100, sequenceid=484, filesize=9.5 K 2023-03-30 16:55:57,426 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 4038776cda6db2eaa29896eab9262a79 in 1304ms, sequenceid=484, compaction requested=true 2023-03-30 16:55:57,427 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:57,427 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:57,427 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:57,427 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:57,427 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,428 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:57,428 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:57,431 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:57,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:57,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a59328b[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:57,433 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:57,433 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:57,434 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:57,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:57,435 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,435 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:57,435 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:57,435 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:57,435 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:57,437 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:57,437 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:57,437 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:57,437 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:57,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:57,442 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:57,442 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:57,442 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:57,442 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:57,442 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:57,444 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:57,444 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:57,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:57,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:57,445 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:57,446 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,448 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,457 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,457 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:57,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:57,457 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:57,457 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:57,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:57,457 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:57,457 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,457 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:57,457 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:57,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:57,457 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:57,458 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:57,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:57,458 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:57,458 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:57,558 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:57,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:57,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:57,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:57,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:57,565 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:57,565 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:57,565 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:57,565 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:57,565 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:57,565 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:57,567 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:57,567 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:57,567 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:57,567 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:57,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:57,568 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:57,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:57,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:57,569 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:57,569 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:57,569 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:57,569 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:57,569 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-03-30 16:55:57,569 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:57,569 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:57,569 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:57,569 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:57,569 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:57,569 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:57,990 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/163c6049b20c461a84d3233c48855c02 2023-03-30 16:55:58,000 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/bc603688d6f24f408e9b5fb1a7325f27 2023-03-30 16:55:58,010 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/dc73e09f3da446cab662d542be247ebb 2023-03-30 16:55:58,014 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/163c6049b20c461a84d3233c48855c02 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/163c6049b20c461a84d3233c48855c02 2023-03-30 16:55:58,018 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/163c6049b20c461a84d3233c48855c02, entries=150, sequenceid=518, filesize=11.9 K 2023-03-30 16:55:58,018 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/bc603688d6f24f408e9b5fb1a7325f27 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/bc603688d6f24f408e9b5fb1a7325f27 2023-03-30 16:55:58,021 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/bc603688d6f24f408e9b5fb1a7325f27, entries=150, sequenceid=518, filesize=11.9 K 2023-03-30 16:55:58,022 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/dc73e09f3da446cab662d542be247ebb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/dc73e09f3da446cab662d542be247ebb 2023-03-30 16:55:58,026 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/dc73e09f3da446cab662d542be247ebb, entries=150, sequenceid=518, filesize=11.9 K 2023-03-30 16:55:58,027 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=0 B/0 for 4038776cda6db2eaa29896eab9262a79 in 458ms, sequenceid=518, compaction requested=true 2023-03-30 16:55:58,027 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:58,027 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:58,027 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:58,027 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:58,027 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,027 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:58,027 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:58,029 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:58,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:58,029 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,030 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:58,030 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:58,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:58,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:58,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:58,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@715c410e[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:58,031 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:58,031 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,032 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,032 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:58,032 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:58,032 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:58,034 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:58,035 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,035 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:58,035 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:58,035 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:58,035 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:58,035 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:58,035 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:58,035 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:58,035 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:58,035 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:58,036 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,036 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:58,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,036 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:58,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,037 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,039 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:58,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,054 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,054 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:58,054 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:58,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:58,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:58,054 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:58,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:58,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:58,054 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,054 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,054 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,054 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,054 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,054 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:58,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:58,054 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:58,054 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:58,155 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:58,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:58,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:58,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:58,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:58,158 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:58,158 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:58,158 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:58,158 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:58,158 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,158 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,160 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,160 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:58,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:58,160 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,160 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:58,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:58,163 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:58,163 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:58,163 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:58,168 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:58,168 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:58,168 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:58,168 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:58,168 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:58,168 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:58,168 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:58,168 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:58,168 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,168 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:58,168 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:58,171 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,171 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:58,171 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,171 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:58,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:58,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73706d4b[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:58,173 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:58,173 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,174 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,175 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,175 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:58,175 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:58,175 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:58,175 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:58,176 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:58,177 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:58,177 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,177 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:58,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:58,180 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:58,180 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:58,180 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:58,180 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:58,181 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:58,183 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:58,183 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,183 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:58,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:58,183 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:58,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:58,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:58,185 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:58,189 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:58,189 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:58,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:58,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:58,189 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:58,189 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:58,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:58,189 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,189 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,190 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,190 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,190 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,190 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,190 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:58,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:58,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:58,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:58,190 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:58,290 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:58,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:58,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:58,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:58,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:58,297 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:58,297 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:58,297 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:58,297 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:58,298 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,298 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,299 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:58,299 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:58,300 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,300 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:58,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:58,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:58,308 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:58,308 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:58,308 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:58,308 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:58,308 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:58,308 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:58,308 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:58,309 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:58,309 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:58,309 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:58,309 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:58,309 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,309 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:58,309 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:58,310 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,310 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:58,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:58,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:58,311 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,311 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:58,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:58,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:58,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:58,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21cd3009[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:58,313 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:58,313 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,315 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,315 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:58,315 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:58,315 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:58,316 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,316 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:58,317 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:58,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:58,317 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:58,317 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:58,317 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:58,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:58,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:58,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:58,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:58,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:58,321 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:58,321 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:58,321 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:58,321 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:58,321 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:58,322 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,322 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,322 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:58,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:58,323 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,324 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,326 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:58,327 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:58,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:58,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:58,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:58,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:58,327 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,327 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,327 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,327 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,327 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,327 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,327 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:58,327 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:58,327 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:58,327 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:58,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:58,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:55:58,381 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:55:58,382 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:58,382 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:58,382 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:58,382 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:58,382 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:58,382 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:55:58,427 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:58,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:58,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:58,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:58,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:58,431 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:58,431 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:58,432 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:58,432 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:58,432 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,432 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195358414, max timestamp=1680195358433Num uniques 859; 2023-03-30 16:55:58,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:55:58,435 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,435 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:58,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:58,435 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,435 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:58,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195358414, max timestamp=1680195358433Num uniques 859; 2023-03-30 16:55:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:55:58,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195358414, max timestamp=1680195358433Num uniques 859; 2023-03-30 16:55:58,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:55:58,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:58,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:58,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:58,436 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:58,436 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:58,436 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:58,437 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:58,437 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:58,437 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:58,437 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:58,437 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:58,437 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:58,437 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:58,437 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:58,437 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,437 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:58,437 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:58,438 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,438 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:58,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:58,439 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,439 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:58,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:58,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:58,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:58,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:58,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62c393d1[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:58,440 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:58,440 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,442 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,442 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:58,442 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:58,442 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:58,442 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,442 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:58,443 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:58,443 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,443 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:58,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:58,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:58,443 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:58,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:58,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:58,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:58,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:58,447 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:58,447 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:58,447 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:58,447 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:58,447 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:58,450 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,450 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:58,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,450 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:58,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:58,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,451 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:58,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,453 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:58,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,456 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,456 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,456 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:58,456 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,456 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:58,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:58,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:58,456 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:58,456 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:58,456 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,456 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:58,457 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:58,456 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,457 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:58,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:58,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:58,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195418474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:58,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195418478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:58,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195418479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:58,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195418487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,557 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:58,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:58,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:58,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:58,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:58,604 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:58,604 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:58,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:58,604 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:58,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195418590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,604 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:58,605 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,605 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,606 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,606 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:58,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:58,607 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,607 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:58,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:58,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:58,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:58,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195418598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:58,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195418600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:58,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195418602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:58,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:58,616 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:58,616 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:58,616 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:58,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:58,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:58,628 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:58,628 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:58,628 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:58,628 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:58,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:58,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:58,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:58,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:58,641 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,641 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:58,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:58,642 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,643 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:58,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:58,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7104eab6[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:58,644 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:58,644 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,645 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,646 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:58,646 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:58,646 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:58,646 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,646 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:58,647 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:58,647 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:58,647 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:58,647 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:58,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:58,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:58,653 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:58,653 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:58,653 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:58,653 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:58,653 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:58,655 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,655 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:58,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,655 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:58,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:58,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:58,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:58,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,656 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,668 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,674 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:58,674 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:58,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:58,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:58,684 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,684 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,684 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:58,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:58,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:58,684 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,684 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,684 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,684 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,684 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,684 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:58,684 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:58,784 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:58,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:58,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:58,788 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/21c0caa4240741418e3d88e009e60c1b 2023-03-30 16:55:58,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:58,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:58,800 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:58,800 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:58,800 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:58,800 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:58,801 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,801 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,802 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:58,802 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:58,802 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,803 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:58,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:58,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:58,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:58,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:58,812 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:58,812 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:58,812 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:58,813 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:58,813 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:58,813 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:58,813 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:58,813 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:58,813 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:58,813 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:58,814 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:58,814 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,814 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:58,814 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:58,815 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:58,815 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:58,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:58,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16ecbe8[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:58,817 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:58,817 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:58,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195418808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,819 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:58,819 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,819 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:58,822 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,822 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:58,822 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:58,822 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:58,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:58,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:58,826 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:58,827 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:58,827 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:58,827 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:58,827 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:58,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:58,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195418813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:58,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195418814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:58,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195418814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,828 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:58,829 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,829 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:58,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:58,829 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,830 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:58,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,832 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:58,832 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:58,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:58,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:58,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:58,833 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:58,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:58,833 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,833 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,833 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,833 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,833 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,833 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,833 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,833 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:58,833 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:58,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:58,835 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/0f1f20fcd9da45aa876969427abeea00 2023-03-30 16:55:58,933 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:58,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:58,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:58,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:58,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:58,937 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:58,937 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:58,937 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:58,937 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:58,937 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,937 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,938 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:58,938 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:58,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:58,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,939 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,939 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:58,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:58,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:58,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:58,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:58,940 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:58,940 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:58,940 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:58,940 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:58,940 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:58,940 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:58,940 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:58,940 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:58,940 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:58,940 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:58,940 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:58,940 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,940 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:58,941 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:58,943 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,943 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:58,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:58,943 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:58,944 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:58,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:58,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:58,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@496bfeb4[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:58,949 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:58,949 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,951 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,951 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:58,951 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,951 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:58,951 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:58,951 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:58,952 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,952 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:58,952 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:58,952 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:58,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:58,954 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:58,954 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:58,954 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:58,954 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:58,954 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:58,956 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,956 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:58,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,956 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:58,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:58,956 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,957 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,961 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,961 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:58,961 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:58,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:58,961 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:58,961 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:58,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:58,961 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:58,961 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:58,962 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:58,962 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,962 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:58,962 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:58,962 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:58,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:59,062 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:59,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:59,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:59,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:59,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:59,065 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:59,065 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:59,065 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:59,065 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:59,065 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,065 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:59,066 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,066 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:59,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:59,067 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,067 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:59,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:59,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:59,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:59,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:59,068 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:59,068 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:59,068 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:59,068 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:59,070 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,070 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:59,070 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,070 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:59,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:59,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:59,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37c8b153[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:59,071 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:59,071 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,076 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,076 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:59,076 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:59,076 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:59,076 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,076 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:59,078 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,078 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:59,078 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:59,078 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:59,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:59,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:59,086 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:59,086 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:59,086 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:59,086 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:59,086 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:59,088 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,088 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:59,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,088 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:59,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:59,088 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,089 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,091 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:59,091 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,091 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:59,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:59,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:59,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:59,091 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:59,092 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:59,092 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:59,092 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:59,092 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:59,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:59,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:59,092 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,092 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,092 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,092 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:59,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57194 deadline: 1680195419120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:59,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57278 deadline: 1680195419128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:59,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57252 deadline: 1680195419129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:55:59,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57280 deadline: 1680195419130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,192 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:59,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:59,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:59,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:59,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:59,195 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:59,195 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:59,195 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:59,195 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:59,196 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,196 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:59,198 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:59,198 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:59,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:59,198 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,198 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:59,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:59,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:59,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:59,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:59,199 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:59,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:59,199 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:59,199 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:59,199 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:59,199 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:59,199 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:59,199 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:59,200 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:59,200 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. as already flushing 2023-03-30 16:55:59,200 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:59,200 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:59,200 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:59,200 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,200 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:59,200 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:59,201 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,201 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:59,201 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,201 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:59,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:59,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61cfac46[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:59,203 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:59,203 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,204 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,204 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:59,204 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,204 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:59,204 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:59,204 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:59,205 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,206 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:59,206 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:59,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,206 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:59,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:59,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:59,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:59,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:59,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:59,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:59,208 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:59,208 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:59,208 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:59,208 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:59,208 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:59,210 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,210 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,210 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:59,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:59,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:59,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:59,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:59,211 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:59,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,212 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,214 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:59,214 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,214 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:59,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:59,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:59,214 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:59,214 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:59,214 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,214 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:59,214 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,214 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,214 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:59,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:59,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:59,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:59,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:59,215 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:59,215 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:59,271 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/ef8e7c42545d42d29162b98bce7705c4 2023-03-30 16:55:59,275 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/21c0caa4240741418e3d88e009e60c1b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/21c0caa4240741418e3d88e009e60c1b 2023-03-30 16:55:59,278 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/21c0caa4240741418e3d88e009e60c1b, entries=300, sequenceid=530, filesize=19.0 K 2023-03-30 16:55:59,279 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/0f1f20fcd9da45aa876969427abeea00 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/0f1f20fcd9da45aa876969427abeea00 2023-03-30 16:55:59,282 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/0f1f20fcd9da45aa876969427abeea00, entries=150, sequenceid=530, filesize=11.9 K 2023-03-30 16:55:59,283 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/ef8e7c42545d42d29162b98bce7705c4 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ef8e7c42545d42d29162b98bce7705c4 2023-03-30 16:55:59,287 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ef8e7c42545d42d29162b98bce7705c4, entries=150, sequenceid=530, filesize=11.9 K 2023-03-30 16:55:59,288 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 4038776cda6db2eaa29896eab9262a79 in 907ms, sequenceid=530, compaction requested=true 2023-03-30 16:55:59,288 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:59,288 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:59,288 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-30 16:55:59,288 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-30 16:55:59,288 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:59,288 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:59,292 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 95944 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-30 16:55:59,292 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/A is initiating minor compaction (all files) 2023-03-30 16:55:59,292 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83744 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-30 16:55:59,292 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/A in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:59,292 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/B is initiating minor compaction (all files) 2023-03-30 16:55:59,292 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/B in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:59,292 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/74092bd1cbf64ca1ac9bba341b572386, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/f40907c483e94b46a1f7b56527f3c200, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/239d3dfa88614696adebb1ab8c4a9d3f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/7233dbe2aee84c268a7738800b8d43ba, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/69a034d1bf174d6e983d6b7002fe1e2c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/163c6049b20c461a84d3233c48855c02, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/21c0caa4240741418e3d88e009e60c1b] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=93.7 K 2023-03-30 16:55:59,292 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/514e98c2d7544119b001bf914a5cb63f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/11a48877bd784c10a91bc7079d99e3be, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/3e46f84b8dd646078c7ca9b72f5af59d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/56bbfbb8c3dd4f9ba36628ab9718e8a6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/2517ec3e65994aab9dd0f433e1ed3a7e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/bc603688d6f24f408e9b5fb1a7325f27, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/0f1f20fcd9da45aa876969427abeea00] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=81.8 K 2023-03-30 16:55:59,292 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 74092bd1cbf64ca1ac9bba341b572386, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=417, earliestPutTs=1680195353936 2023-03-30 16:55:59,293 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 514e98c2d7544119b001bf914a5cb63f, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=417, earliestPutTs=1680195353936 2023-03-30 16:55:59,293 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting f40907c483e94b46a1f7b56527f3c200, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1680195353974 2023-03-30 16:55:59,293 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 11a48877bd784c10a91bc7079d99e3be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1680195353974 2023-03-30 16:55:59,293 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 239d3dfa88614696adebb1ab8c4a9d3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1680195353998 2023-03-30 16:55:59,294 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 3e46f84b8dd646078c7ca9b72f5af59d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1680195353998 2023-03-30 16:55:59,294 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 7233dbe2aee84c268a7738800b8d43ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=472, earliestPutTs=1680195354666 2023-03-30 16:55:59,294 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 56bbfbb8c3dd4f9ba36628ab9718e8a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=472, earliestPutTs=1680195354666 2023-03-30 16:55:59,294 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 69a034d1bf174d6e983d6b7002fe1e2c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680195355833 2023-03-30 16:55:59,295 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 2517ec3e65994aab9dd0f433e1ed3a7e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680195355833 2023-03-30 16:55:59,295 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 163c6049b20c461a84d3233c48855c02, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680195356217 2023-03-30 16:55:59,295 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting bc603688d6f24f408e9b5fb1a7325f27, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680195356217 2023-03-30 16:55:59,295 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 21c0caa4240741418e3d88e009e60c1b, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=530, earliestPutTs=1680195358373 2023-03-30 16:55:59,295 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 0f1f20fcd9da45aa876969427abeea00, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=530, earliestPutTs=1680195358377 2023-03-30 16:55:59,313 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#B#compaction#450 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:59,314 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#A#compaction#449 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:55:59,315 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:59,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:59,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:59,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:59,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:59,324 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:59,324 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:59,324 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:59,324 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:59,325 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,325 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:59,326 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,326 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:59,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:59,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:59,326 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,326 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:59,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:59,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:59,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:59,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:59,327 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:59,327 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:59,327 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:59,328 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:59,328 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:59,328 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:59,328 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:59,328 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-30 16:55:59,328 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:55:59,328 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:59,328 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:55:59,328 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:59,328 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:55:59,328 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:55:59,337 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=566 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/acaaa3158e434059b2ad003c8073e138 2023-03-30 16:55:59,347 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=566 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/3b3c1a893dcd4ae699382b4dc086585d 2023-03-30 16:55:59,370 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=566 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/7e395dc2208b41c7bcaea90ba8fa4341 2023-03-30 16:55:59,374 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/acaaa3158e434059b2ad003c8073e138 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/acaaa3158e434059b2ad003c8073e138 2023-03-30 16:55:59,378 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/acaaa3158e434059b2ad003c8073e138, entries=150, sequenceid=566, filesize=11.9 K 2023-03-30 16:55:59,379 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/3b3c1a893dcd4ae699382b4dc086585d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/3b3c1a893dcd4ae699382b4dc086585d 2023-03-30 16:55:59,383 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/3b3c1a893dcd4ae699382b4dc086585d, entries=150, sequenceid=566, filesize=11.9 K 2023-03-30 16:55:59,383 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/7e395dc2208b41c7bcaea90ba8fa4341 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/7e395dc2208b41c7bcaea90ba8fa4341 2023-03-30 16:55:59,399 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/7e395dc2208b41c7bcaea90ba8fa4341, entries=150, sequenceid=566, filesize=11.9 K 2023-03-30 16:55:59,400 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for 4038776cda6db2eaa29896eab9262a79 in 71ms, sequenceid=566, compaction requested=true 2023-03-30 16:55:59,400 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:59,400 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:59,400 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:59,400 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:59,400 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:59,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:59,405 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,405 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:59,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:59,406 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,406 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:59,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:59,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16890a1c[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:59,407 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:59,407 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,409 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:59,409 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:59,409 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,409 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:59,410 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:59,410 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:59,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:59,413 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:59,413 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:59,413 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:59,413 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:59,413 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:59,415 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,415 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,415 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:59,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:59,415 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:59,417 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:59,420 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:59,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:59,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:59,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:59,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:59,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:59,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:59,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:59,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:59,421 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:59,421 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:59,521 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:59,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:59,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:55:59,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:55:59,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:55:59,524 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:55:59,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:59,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:55:59,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:55:59,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:59,526 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:59,526 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:59,527 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:59,527 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:55:59,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:55:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:59,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:55:59,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:55:59,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:55:59,528 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:55:59,528 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:55:59,528 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:55:59,528 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:55:59,528 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:55:59,529 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:59,529 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. started... 2023-03-30 16:55:59,529 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:59,529 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:59,529 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:55:59,529 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:55:59,529 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,529 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:55:59,529 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:55:59,530 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,530 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:59,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:59,530 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,530 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:55:59,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:59,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:55:59,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d827[Count = 0] remaining members to acquire global barrier 2023-03-30 16:55:59,532 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:55:59,532 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,533 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:55:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:55:59,533 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,533 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:55:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:55:59,542 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:55:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:55:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:55:59,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:55:59,545 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:55:59,545 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:55:59,545 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:55:59,545 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:55:59,545 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:55:59,546 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,546 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:55:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:55:59,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,546 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:59,547 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:55:59,548 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:55:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:55:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,550 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,550 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:55:59,550 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:59,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:55:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:55:59,550 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:55:59,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:55:59,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:55:59,550 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:55:59,550 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:55:59,550 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:55:59,550 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,550 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:55:59,551 DEBUG [Thread-2823] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:55:59,551 DEBUG [Thread-2823] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:55:59,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:55:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:55:59,551 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:55:59,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:55:59,554 DEBUG [Thread-2826] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x028a3a77 to 127.0.0.1:58435 2023-03-30 16:55:59,554 DEBUG [Thread-2826] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:55:59,555 DEBUG [Thread-2832] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78fa3dee to 127.0.0.1:58435 2023-03-30 16:55:59,555 DEBUG [Thread-2832] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:55:59,557 DEBUG [Thread-2824] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50150525 to 127.0.0.1:58435 2023-03-30 16:55:59,557 DEBUG [Thread-2824] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:55:59,560 DEBUG [Thread-2828] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4cefb012 to 127.0.0.1:58435 2023-03-30 16:55:59,560 DEBUG [Thread-2828] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:55:59,560 DEBUG [Thread-2830] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24a917df to 127.0.0.1:58435 2023-03-30 16:55:59,560 DEBUG [Thread-2830] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:55:59,628 DEBUG [Thread-2815] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3fb41c50 to 127.0.0.1:58435 2023-03-30 16:55:59,628 DEBUG [Thread-2815] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:55:59,639 DEBUG [Thread-2821] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32cd672d to 127.0.0.1:58435 2023-03-30 16:55:59,640 DEBUG [Thread-2821] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:55:59,640 DEBUG [Thread-2819] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4539ed0c to 127.0.0.1:58435 2023-03-30 16:55:59,640 DEBUG [Thread-2819] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:55:59,641 DEBUG [Thread-2817] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x357cb237 to 127.0.0.1:58435 2023-03-30 16:55:59,641 DEBUG [Thread-2817] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:55:59,651 DEBUG [Thread-2823] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:55:59,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:55:59,730 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/21efb624d3c6404e9c90c1573c4a068f as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/21efb624d3c6404e9c90c1573c4a068f 2023-03-30 16:55:59,730 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/6003cc2678f64e82a8a050a4fa3bc759 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/6003cc2678f64e82a8a050a4fa3bc759 2023-03-30 16:55:59,734 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/B of 4038776cda6db2eaa29896eab9262a79 into 21efb624d3c6404e9c90c1573c4a068f(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:59,734 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/A of 4038776cda6db2eaa29896eab9262a79 into 6003cc2678f64e82a8a050a4fa3bc759(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:55:59,734 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:59,734 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:55:59,734 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/B, priority=9, startTime=1680195359288; duration=0sec 2023-03-30 16:55:59,734 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/A, priority=9, startTime=1680195359288; duration=0sec 2023-03-30 16:55:59,734 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:55:59,734 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:55:59,735 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-30 16:55:59,736 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95930 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-30 16:55:59,736 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 4038776cda6db2eaa29896eab9262a79/C is initiating minor compaction (all files) 2023-03-30 16:55:59,737 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4038776cda6db2eaa29896eab9262a79/C in TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:55:59,737 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/bc18dafd25ee4bfe803ced03086f2ca9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/5bab6431ab1e48debc03b845906b7685, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/a3cd8ce62b9f44f4a3989a5b8e2902d6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/4fe94ab84c1f47289dd0d4c5c1dce743, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/6962f7188e1b425abde25c6cbb64efec, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/dc73e09f3da446cab662d542be247ebb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ef8e7c42545d42d29162b98bce7705c4, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/7e395dc2208b41c7bcaea90ba8fa4341] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp, totalSize=93.7 K 2023-03-30 16:55:59,737 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting bc18dafd25ee4bfe803ced03086f2ca9, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=417, earliestPutTs=1680195353936 2023-03-30 16:55:59,737 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 5bab6431ab1e48debc03b845906b7685, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1680195353974 2023-03-30 16:55:59,738 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting a3cd8ce62b9f44f4a3989a5b8e2902d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1680195353998 2023-03-30 16:55:59,738 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 4fe94ab84c1f47289dd0d4c5c1dce743, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=472, earliestPutTs=1680195354666 2023-03-30 16:55:59,738 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 6962f7188e1b425abde25c6cbb64efec, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680195355833 2023-03-30 16:55:59,738 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting dc73e09f3da446cab662d542be247ebb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680195356217 2023-03-30 16:55:59,739 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting ef8e7c42545d42d29162b98bce7705c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=530, earliestPutTs=1680195358377 2023-03-30 16:55:59,739 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 7e395dc2208b41c7bcaea90ba8fa4341, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=566, earliestPutTs=1680195358477 2023-03-30 16:55:59,748 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 4038776cda6db2eaa29896eab9262a79#C#compaction#454 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:56:00,155 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/5525119da53d4fc8ba7b44db2af4d584 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/5525119da53d4fc8ba7b44db2af4d584 2023-03-30 16:56:00,158 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 4038776cda6db2eaa29896eab9262a79/C of 4038776cda6db2eaa29896eab9262a79 into 5525119da53d4fc8ba7b44db2af4d584(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:56:00,158 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:56:00,158 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79., storeName=4038776cda6db2eaa29896eab9262a79/C, priority=8, startTime=1680195359288; duration=0sec 2023-03-30 16:56:00,158 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:56:04,157 DEBUG [Thread-2813] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c8fbc35 to 127.0.0.1:58435 2023-03-30 16:56:04,157 DEBUG [Thread-2813] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:56:04,158 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-30 16:56:04,158 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 67 2023-03-30 16:56:04,158 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 94 2023-03-30 16:56:04,158 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 99 2023-03-30 16:56:04,158 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 104 2023-03-30 16:56:04,158 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 101 2023-03-30 16:56:04,158 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-30 16:56:04,158 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-30 16:56:04,158 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(402): scanned 1889 2023-03-30 16:56:04,158 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(403): verified 5664 rows 2023-03-30 16:56:04,158 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(402): scanned 1906 2023-03-30 16:56:04,158 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(403): verified 5717 rows 2023-03-30 16:56:04,158 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(402): scanned 1912 2023-03-30 16:56:04,158 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(403): verified 5728 rows 2023-03-30 16:56:04,158 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(402): scanned 1911 2023-03-30 16:56:04,158 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(403): verified 5728 rows 2023-03-30 16:56:04,158 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(402): scanned 1919 2023-03-30 16:56:04,158 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(403): verified 5753 rows 2023-03-30 16:56:04,158 INFO [Listener at localhost/38075] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-30 16:56:04,158 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77d7b49f to 127.0.0.1:58435 2023-03-30 16:56:04,159 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:56:04,161 INFO [Listener at localhost/38075] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-30 16:56:04,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-30 16:56:04,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-30 16:56:04,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-30 16:56:04,166 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195364166"}]},"ts":"1680195364166"} 2023-03-30 16:56:04,168 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-30 16:56:04,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-30 16:56:04,363 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-30 16:56:04,364 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4038776cda6db2eaa29896eab9262a79, UNASSIGN}] 2023-03-30 16:56:04,366 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4038776cda6db2eaa29896eab9262a79, UNASSIGN 2023-03-30 16:56:04,366 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=4038776cda6db2eaa29896eab9262a79, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:04,366 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195364366"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680195364366"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680195364366"}]},"ts":"1680195364366"} 2023-03-30 16:56:04,368 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285}] 2023-03-30 16:56:04,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-30 16:56:04,520 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:56:04,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4038776cda6db2eaa29896eab9262a79, disabling compactions & flushes 2023-03-30 16:56:04,520 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:56:04,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:56:04,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. after waiting 0 ms 2023-03-30 16:56:04,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:56:04,520 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 4038776cda6db2eaa29896eab9262a79 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-30 16:56:04,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=A 2023-03-30 16:56:04,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:04,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=B 2023-03-30 16:56:04,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:04,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4038776cda6db2eaa29896eab9262a79, store=C 2023-03-30 16:56:04,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:04,526 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/d442988adaca40c384eee839a8725d9b 2023-03-30 16:56:04,535 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/4deecc1daac54ceeab1017db7545242b 2023-03-30 16:56:04,545 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/b7554de675204ede88981dc9e27d75d9 2023-03-30 16:56:04,549 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/A/d442988adaca40c384eee839a8725d9b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/d442988adaca40c384eee839a8725d9b 2023-03-30 16:56:04,552 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/d442988adaca40c384eee839a8725d9b, entries=150, sequenceid=578, filesize=11.9 K 2023-03-30 16:56:04,552 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/B/4deecc1daac54ceeab1017db7545242b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/4deecc1daac54ceeab1017db7545242b 2023-03-30 16:56:04,556 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/4deecc1daac54ceeab1017db7545242b, entries=150, sequenceid=578, filesize=11.9 K 2023-03-30 16:56:04,556 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/.tmp/C/b7554de675204ede88981dc9e27d75d9 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/b7554de675204ede88981dc9e27d75d9 2023-03-30 16:56:04,560 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/b7554de675204ede88981dc9e27d75d9, entries=150, sequenceid=578, filesize=11.9 K 2023-03-30 16:56:04,560 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 4038776cda6db2eaa29896eab9262a79 in 40ms, sequenceid=578, compaction requested=true 2023-03-30 16:56:04,586 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/b640b22151534e0186ba76da6ca56a63, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/75574a42e1b94f4db62b4b474a8488bf, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/ec6c20b1db7e4451890454e409970902, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/a35125cf1c424f879d9640edaab88071, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/bed124d544914fdb9ee870765f1048dc, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/1d529e778f9a40f08f49bce3609b0710, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/de0bd5ac988b4a278737c4ec7f9b9022, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/2e0fccbb235945bc82ac48ba455b65e6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/b596abb4179b45b3ac5d03d127113c3e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/cc765ee7d6fe423da2e20cc574584132, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/7e5e451433864e049670c113afa2a5f5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/3810673bf94e46458cbb5a9237030ccb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/7f59055f532a48308226c63f879d6997, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/eb285f1e7b724a0dace42da0199d18c6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/5443d9ece19e4a138c2655229ec41bc8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/1a65f3e079d14bb9a46dab9a5fad2d82, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/e35878dea0e7430c8234c61dee9c8603, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/f89e7a7348134ae481388deeeda727f8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/9c18d9fc1b644437b10692954e6ad3a6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/5b181c2e0a864813b088aa4a000735ae, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/e1e0da1d84ae42a1aaa31bc87ef3715c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/70c5653ac0a1488a87ebc147154109d1, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/b8a6f112dc8c4c9ebe19fc11a424702a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/e8f27cfe61f14baea1508ace7908a9b3, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/69d0bd42070843b2a8c19a115fe7806b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/4e91b2997b934d92947b95dc61597f9a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/74092bd1cbf64ca1ac9bba341b572386, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/f40907c483e94b46a1f7b56527f3c200, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/239d3dfa88614696adebb1ab8c4a9d3f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/7233dbe2aee84c268a7738800b8d43ba, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/69a034d1bf174d6e983d6b7002fe1e2c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/163c6049b20c461a84d3233c48855c02, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/21c0caa4240741418e3d88e009e60c1b] to archive 2023-03-30 16:56:04,587 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 16:56:04,589 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/b640b22151534e0186ba76da6ca56a63 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/b640b22151534e0186ba76da6ca56a63 2023-03-30 16:56:04,590 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/75574a42e1b94f4db62b4b474a8488bf to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/75574a42e1b94f4db62b4b474a8488bf 2023-03-30 16:56:04,591 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/ec6c20b1db7e4451890454e409970902 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/ec6c20b1db7e4451890454e409970902 2023-03-30 16:56:04,592 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/a35125cf1c424f879d9640edaab88071 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/a35125cf1c424f879d9640edaab88071 2023-03-30 16:56:04,593 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/bed124d544914fdb9ee870765f1048dc to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/bed124d544914fdb9ee870765f1048dc 2023-03-30 16:56:04,594 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/1d529e778f9a40f08f49bce3609b0710 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/1d529e778f9a40f08f49bce3609b0710 2023-03-30 16:56:04,595 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/de0bd5ac988b4a278737c4ec7f9b9022 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/de0bd5ac988b4a278737c4ec7f9b9022 2023-03-30 16:56:04,596 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/2e0fccbb235945bc82ac48ba455b65e6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/2e0fccbb235945bc82ac48ba455b65e6 2023-03-30 16:56:04,597 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/b596abb4179b45b3ac5d03d127113c3e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/b596abb4179b45b3ac5d03d127113c3e 2023-03-30 16:56:04,599 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/cc765ee7d6fe423da2e20cc574584132 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/cc765ee7d6fe423da2e20cc574584132 2023-03-30 16:56:04,600 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/7e5e451433864e049670c113afa2a5f5 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/7e5e451433864e049670c113afa2a5f5 2023-03-30 16:56:04,601 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/3810673bf94e46458cbb5a9237030ccb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/3810673bf94e46458cbb5a9237030ccb 2023-03-30 16:56:04,602 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/7f59055f532a48308226c63f879d6997 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/7f59055f532a48308226c63f879d6997 2023-03-30 16:56:04,603 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/eb285f1e7b724a0dace42da0199d18c6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/eb285f1e7b724a0dace42da0199d18c6 2023-03-30 16:56:04,604 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/5443d9ece19e4a138c2655229ec41bc8 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/5443d9ece19e4a138c2655229ec41bc8 2023-03-30 16:56:04,605 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/1a65f3e079d14bb9a46dab9a5fad2d82 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/1a65f3e079d14bb9a46dab9a5fad2d82 2023-03-30 16:56:04,606 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/e35878dea0e7430c8234c61dee9c8603 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/e35878dea0e7430c8234c61dee9c8603 2023-03-30 16:56:04,607 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/f89e7a7348134ae481388deeeda727f8 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/f89e7a7348134ae481388deeeda727f8 2023-03-30 16:56:04,609 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/9c18d9fc1b644437b10692954e6ad3a6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/9c18d9fc1b644437b10692954e6ad3a6 2023-03-30 16:56:04,610 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/5b181c2e0a864813b088aa4a000735ae to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/5b181c2e0a864813b088aa4a000735ae 2023-03-30 16:56:04,611 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/e1e0da1d84ae42a1aaa31bc87ef3715c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/e1e0da1d84ae42a1aaa31bc87ef3715c 2023-03-30 16:56:04,612 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/70c5653ac0a1488a87ebc147154109d1 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/70c5653ac0a1488a87ebc147154109d1 2023-03-30 16:56:04,613 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/b8a6f112dc8c4c9ebe19fc11a424702a to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/b8a6f112dc8c4c9ebe19fc11a424702a 2023-03-30 16:56:04,614 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/e8f27cfe61f14baea1508ace7908a9b3 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/e8f27cfe61f14baea1508ace7908a9b3 2023-03-30 16:56:04,615 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/69d0bd42070843b2a8c19a115fe7806b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/69d0bd42070843b2a8c19a115fe7806b 2023-03-30 16:56:04,616 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/4e91b2997b934d92947b95dc61597f9a to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/4e91b2997b934d92947b95dc61597f9a 2023-03-30 16:56:04,617 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/74092bd1cbf64ca1ac9bba341b572386 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/74092bd1cbf64ca1ac9bba341b572386 2023-03-30 16:56:04,618 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/f40907c483e94b46a1f7b56527f3c200 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/f40907c483e94b46a1f7b56527f3c200 2023-03-30 16:56:04,619 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/239d3dfa88614696adebb1ab8c4a9d3f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/239d3dfa88614696adebb1ab8c4a9d3f 2023-03-30 16:56:04,621 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/7233dbe2aee84c268a7738800b8d43ba to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/7233dbe2aee84c268a7738800b8d43ba 2023-03-30 16:56:04,622 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/69a034d1bf174d6e983d6b7002fe1e2c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/69a034d1bf174d6e983d6b7002fe1e2c 2023-03-30 16:56:04,623 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/163c6049b20c461a84d3233c48855c02 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/163c6049b20c461a84d3233c48855c02 2023-03-30 16:56:04,624 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/21c0caa4240741418e3d88e009e60c1b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/21c0caa4240741418e3d88e009e60c1b 2023-03-30 16:56:04,628 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/fe0353d26bab450b9c69e02f6ef558c2, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/43f4293fb14140548bf6bfaa0cdd2e1a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/5685f49a3f82405bad04127e8d9985ac, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/6988fcd6629d47be9487356258e156a6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/6ba3d430cfe3496a9edddc1600c24892, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/74e38e865078432ca2578c1454cba9ad, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/94ef97fac59144d0ade6d1f7bbe22b40, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/9a30d6f5b06c4bed837c13b0b367f508, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/305ed549fa09479ea6f1e5838a3411a9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/4ac7e696e31b4c34807cd1eeb7e323cb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/e756af1ce0c14e2d9f8cb698daa92518, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/f4610c52a8474e31b888220c2a6011ad, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/48035ff5d48f4cf7b99f995e9b8106eb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/6a49280309724c66978fcfabbe57d849, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/385a6f9105f648ffae0e4bff001a1b70, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/9fe1879d858149e6aede4670b393df75, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/010bfe1f53d54c4787bac67092a8430d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/3a7060e7ceba45a893c0933554c843a5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/4f0ff42414e54766a02e13decaf87f03, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/4bab7b5724d646c38aa3c985607bc124, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/e974090b7b6b47f492548b9309adeebd, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/83d3915746b64cea84dd9fc8950a2e47, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/ac2a5302097d4a82badad9d394f79f1a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/0cae7d54490441379e7ec8837e55e25c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/7c4941338a4f46e4813d6192148fe38f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/514e98c2d7544119b001bf914a5cb63f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/437757916c014c8eaf377a8b69feb596, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/11a48877bd784c10a91bc7079d99e3be, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/3e46f84b8dd646078c7ca9b72f5af59d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/56bbfbb8c3dd4f9ba36628ab9718e8a6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/2517ec3e65994aab9dd0f433e1ed3a7e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/bc603688d6f24f408e9b5fb1a7325f27, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/0f1f20fcd9da45aa876969427abeea00] to archive 2023-03-30 16:56:04,628 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 16:56:04,632 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/fe0353d26bab450b9c69e02f6ef558c2 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/fe0353d26bab450b9c69e02f6ef558c2 2023-03-30 16:56:04,633 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/43f4293fb14140548bf6bfaa0cdd2e1a to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/43f4293fb14140548bf6bfaa0cdd2e1a 2023-03-30 16:56:04,634 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/5685f49a3f82405bad04127e8d9985ac to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/5685f49a3f82405bad04127e8d9985ac 2023-03-30 16:56:04,635 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/6988fcd6629d47be9487356258e156a6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/6988fcd6629d47be9487356258e156a6 2023-03-30 16:56:04,636 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/6ba3d430cfe3496a9edddc1600c24892 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/6ba3d430cfe3496a9edddc1600c24892 2023-03-30 16:56:04,637 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/74e38e865078432ca2578c1454cba9ad to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/74e38e865078432ca2578c1454cba9ad 2023-03-30 16:56:04,638 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/94ef97fac59144d0ade6d1f7bbe22b40 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/94ef97fac59144d0ade6d1f7bbe22b40 2023-03-30 16:56:04,639 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/9a30d6f5b06c4bed837c13b0b367f508 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/9a30d6f5b06c4bed837c13b0b367f508 2023-03-30 16:56:04,640 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/305ed549fa09479ea6f1e5838a3411a9 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/305ed549fa09479ea6f1e5838a3411a9 2023-03-30 16:56:04,641 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/4ac7e696e31b4c34807cd1eeb7e323cb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/4ac7e696e31b4c34807cd1eeb7e323cb 2023-03-30 16:56:04,643 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/e756af1ce0c14e2d9f8cb698daa92518 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/e756af1ce0c14e2d9f8cb698daa92518 2023-03-30 16:56:04,644 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/f4610c52a8474e31b888220c2a6011ad to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/f4610c52a8474e31b888220c2a6011ad 2023-03-30 16:56:04,645 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/48035ff5d48f4cf7b99f995e9b8106eb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/48035ff5d48f4cf7b99f995e9b8106eb 2023-03-30 16:56:04,646 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/6a49280309724c66978fcfabbe57d849 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/6a49280309724c66978fcfabbe57d849 2023-03-30 16:56:04,647 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/385a6f9105f648ffae0e4bff001a1b70 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/385a6f9105f648ffae0e4bff001a1b70 2023-03-30 16:56:04,648 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/9fe1879d858149e6aede4670b393df75 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/9fe1879d858149e6aede4670b393df75 2023-03-30 16:56:04,648 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/010bfe1f53d54c4787bac67092a8430d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/010bfe1f53d54c4787bac67092a8430d 2023-03-30 16:56:04,649 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/3a7060e7ceba45a893c0933554c843a5 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/3a7060e7ceba45a893c0933554c843a5 2023-03-30 16:56:04,650 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/4f0ff42414e54766a02e13decaf87f03 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/4f0ff42414e54766a02e13decaf87f03 2023-03-30 16:56:04,651 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/4bab7b5724d646c38aa3c985607bc124 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/4bab7b5724d646c38aa3c985607bc124 2023-03-30 16:56:04,652 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/e974090b7b6b47f492548b9309adeebd to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/e974090b7b6b47f492548b9309adeebd 2023-03-30 16:56:04,653 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/83d3915746b64cea84dd9fc8950a2e47 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/83d3915746b64cea84dd9fc8950a2e47 2023-03-30 16:56:04,654 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/ac2a5302097d4a82badad9d394f79f1a to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/ac2a5302097d4a82badad9d394f79f1a 2023-03-30 16:56:04,654 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/0cae7d54490441379e7ec8837e55e25c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/0cae7d54490441379e7ec8837e55e25c 2023-03-30 16:56:04,655 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/7c4941338a4f46e4813d6192148fe38f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/7c4941338a4f46e4813d6192148fe38f 2023-03-30 16:56:04,656 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/514e98c2d7544119b001bf914a5cb63f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/514e98c2d7544119b001bf914a5cb63f 2023-03-30 16:56:04,657 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/437757916c014c8eaf377a8b69feb596 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/437757916c014c8eaf377a8b69feb596 2023-03-30 16:56:04,657 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/11a48877bd784c10a91bc7079d99e3be to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/11a48877bd784c10a91bc7079d99e3be 2023-03-30 16:56:04,658 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/3e46f84b8dd646078c7ca9b72f5af59d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/3e46f84b8dd646078c7ca9b72f5af59d 2023-03-30 16:56:04,659 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/56bbfbb8c3dd4f9ba36628ab9718e8a6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/56bbfbb8c3dd4f9ba36628ab9718e8a6 2023-03-30 16:56:04,660 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/2517ec3e65994aab9dd0f433e1ed3a7e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/2517ec3e65994aab9dd0f433e1ed3a7e 2023-03-30 16:56:04,661 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/bc603688d6f24f408e9b5fb1a7325f27 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/bc603688d6f24f408e9b5fb1a7325f27 2023-03-30 16:56:04,662 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/0f1f20fcd9da45aa876969427abeea00 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/0f1f20fcd9da45aa876969427abeea00 2023-03-30 16:56:04,664 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ad80ee225cae4268add354f6817a6095, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/e179218f8e854620be03f6ecb190db11, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ad3417c1cb274c6699ea42f4a1608257, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/5c493d60f5ee4599b13f8d29c20abae0, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/59500937e31646708ad766f6fd36523c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/7b40eaa9ae374f4b9612f7d9067a4c75, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/8b87ce28138740d292f0f61659dd8efd, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/8cec52281b3e4f31b63029f9432f4f83, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/83f05324ea674aef8dc84e0d7c67c76f, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/73dc51beadc04ef8a7004ebf41001e36, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/fa3b499595084e33bec03085e683ad2a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/7ec7546c7fde4cd199d9cec84e79eda3, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/3d51ffd9dc9c41e8afb92f8545a000be, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/c00095d3f5524344951d1fde496bee3b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ae1dcb6aa9024402928a1849b73f2317, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/0f83127160924439bf06047b5ad8f033, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/5256c400bd664fddaadc7538665f2333, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/f11f183a10e44b4b885c8051c798e2a7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/dc6000db51a54e28a4bed8e4018f3f22, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/12c95776af3945dc96467cc26c24f5bc, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/167b390ce3cc44ecbe33419c1e74b4dd, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/4f9a6ae12f934fb881b0044ff9143f31, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/0052b6e09d53407495a2f8ae0ef96896, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/979fe09249bc407790077e0b7f44659e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/57aaf2ca7f174cc0b46b850a30bcf5dc, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/bc18dafd25ee4bfe803ced03086f2ca9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/46b588791c95473ba06dcf19e070e8b7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/5bab6431ab1e48debc03b845906b7685, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/a3cd8ce62b9f44f4a3989a5b8e2902d6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/4fe94ab84c1f47289dd0d4c5c1dce743, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/6962f7188e1b425abde25c6cbb64efec, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/dc73e09f3da446cab662d542be247ebb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ef8e7c42545d42d29162b98bce7705c4, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/7e395dc2208b41c7bcaea90ba8fa4341] to archive 2023-03-30 16:56:04,665 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 16:56:04,666 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ad80ee225cae4268add354f6817a6095 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ad80ee225cae4268add354f6817a6095 2023-03-30 16:56:04,667 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/e179218f8e854620be03f6ecb190db11 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/e179218f8e854620be03f6ecb190db11 2023-03-30 16:56:04,668 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ad3417c1cb274c6699ea42f4a1608257 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ad3417c1cb274c6699ea42f4a1608257 2023-03-30 16:56:04,669 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/5c493d60f5ee4599b13f8d29c20abae0 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/5c493d60f5ee4599b13f8d29c20abae0 2023-03-30 16:56:04,670 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/59500937e31646708ad766f6fd36523c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/59500937e31646708ad766f6fd36523c 2023-03-30 16:56:04,671 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/7b40eaa9ae374f4b9612f7d9067a4c75 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/7b40eaa9ae374f4b9612f7d9067a4c75 2023-03-30 16:56:04,671 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/8b87ce28138740d292f0f61659dd8efd to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/8b87ce28138740d292f0f61659dd8efd 2023-03-30 16:56:04,672 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/8cec52281b3e4f31b63029f9432f4f83 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/8cec52281b3e4f31b63029f9432f4f83 2023-03-30 16:56:04,673 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/83f05324ea674aef8dc84e0d7c67c76f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/83f05324ea674aef8dc84e0d7c67c76f 2023-03-30 16:56:04,674 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/73dc51beadc04ef8a7004ebf41001e36 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/73dc51beadc04ef8a7004ebf41001e36 2023-03-30 16:56:04,674 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/fa3b499595084e33bec03085e683ad2a to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/fa3b499595084e33bec03085e683ad2a 2023-03-30 16:56:04,675 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/7ec7546c7fde4cd199d9cec84e79eda3 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/7ec7546c7fde4cd199d9cec84e79eda3 2023-03-30 16:56:04,676 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/3d51ffd9dc9c41e8afb92f8545a000be to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/3d51ffd9dc9c41e8afb92f8545a000be 2023-03-30 16:56:04,677 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/c00095d3f5524344951d1fde496bee3b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/c00095d3f5524344951d1fde496bee3b 2023-03-30 16:56:04,677 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ae1dcb6aa9024402928a1849b73f2317 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ae1dcb6aa9024402928a1849b73f2317 2023-03-30 16:56:04,678 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/0f83127160924439bf06047b5ad8f033 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/0f83127160924439bf06047b5ad8f033 2023-03-30 16:56:04,679 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/5256c400bd664fddaadc7538665f2333 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/5256c400bd664fddaadc7538665f2333 2023-03-30 16:56:04,680 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/f11f183a10e44b4b885c8051c798e2a7 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/f11f183a10e44b4b885c8051c798e2a7 2023-03-30 16:56:04,681 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/dc6000db51a54e28a4bed8e4018f3f22 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/dc6000db51a54e28a4bed8e4018f3f22 2023-03-30 16:56:04,681 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/12c95776af3945dc96467cc26c24f5bc to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/12c95776af3945dc96467cc26c24f5bc 2023-03-30 16:56:04,682 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/167b390ce3cc44ecbe33419c1e74b4dd to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/167b390ce3cc44ecbe33419c1e74b4dd 2023-03-30 16:56:04,683 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/4f9a6ae12f934fb881b0044ff9143f31 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/4f9a6ae12f934fb881b0044ff9143f31 2023-03-30 16:56:04,684 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/0052b6e09d53407495a2f8ae0ef96896 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/0052b6e09d53407495a2f8ae0ef96896 2023-03-30 16:56:04,685 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/979fe09249bc407790077e0b7f44659e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/979fe09249bc407790077e0b7f44659e 2023-03-30 16:56:04,685 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/57aaf2ca7f174cc0b46b850a30bcf5dc to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/57aaf2ca7f174cc0b46b850a30bcf5dc 2023-03-30 16:56:04,686 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/bc18dafd25ee4bfe803ced03086f2ca9 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/bc18dafd25ee4bfe803ced03086f2ca9 2023-03-30 16:56:04,687 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/46b588791c95473ba06dcf19e070e8b7 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/46b588791c95473ba06dcf19e070e8b7 2023-03-30 16:56:04,688 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/5bab6431ab1e48debc03b845906b7685 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/5bab6431ab1e48debc03b845906b7685 2023-03-30 16:56:04,689 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/a3cd8ce62b9f44f4a3989a5b8e2902d6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/a3cd8ce62b9f44f4a3989a5b8e2902d6 2023-03-30 16:56:04,690 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/4fe94ab84c1f47289dd0d4c5c1dce743 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/4fe94ab84c1f47289dd0d4c5c1dce743 2023-03-30 16:56:04,691 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/6962f7188e1b425abde25c6cbb64efec to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/6962f7188e1b425abde25c6cbb64efec 2023-03-30 16:56:04,691 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/dc73e09f3da446cab662d542be247ebb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/dc73e09f3da446cab662d542be247ebb 2023-03-30 16:56:04,692 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ef8e7c42545d42d29162b98bce7705c4 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/ef8e7c42545d42d29162b98bce7705c4 2023-03-30 16:56:04,693 DEBUG [StoreCloser-TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/7e395dc2208b41c7bcaea90ba8fa4341 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/7e395dc2208b41c7bcaea90ba8fa4341 2023-03-30 16:56:04,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/recovered.edits/581.seqid, newMaxSeqId=581, maxSeqId=1 2023-03-30 16:56:04,698 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79. 2023-03-30 16:56:04,698 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4038776cda6db2eaa29896eab9262a79: 2023-03-30 16:56:04,699 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4038776cda6db2eaa29896eab9262a79 2023-03-30 16:56:04,700 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=4038776cda6db2eaa29896eab9262a79, regionState=CLOSED 2023-03-30 16:56:04,700 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195364700"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680195364700"}]},"ts":"1680195364700"} 2023-03-30 16:56:04,702 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-03-30 16:56:04,702 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 4038776cda6db2eaa29896eab9262a79, server=jenkins-hbase4.apache.org,45193,1680195214285 in 333 msec 2023-03-30 16:56:04,703 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-03-30 16:56:04,703 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4038776cda6db2eaa29896eab9262a79, UNASSIGN in 338 msec 2023-03-30 16:56:04,703 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195364703"}]},"ts":"1680195364703"} 2023-03-30 16:56:04,704 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-30 16:56:04,706 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-30 16:56:04,707 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 546 msec 2023-03-30 16:56:04,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-30 16:56:04,769 INFO [Listener at localhost/38075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-03-30 16:56:04,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-30 16:56:04,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:56:04,772 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:56:04,773 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:56:04,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-30 16:56:04,776 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79 2023-03-30 16:56:04,778 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A, FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B, FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C, FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/recovered.edits] 2023-03-30 16:56:04,780 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/6003cc2678f64e82a8a050a4fa3bc759 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/6003cc2678f64e82a8a050a4fa3bc759 2023-03-30 16:56:04,781 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/acaaa3158e434059b2ad003c8073e138 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/acaaa3158e434059b2ad003c8073e138 2023-03-30 16:56:04,782 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/d442988adaca40c384eee839a8725d9b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/A/d442988adaca40c384eee839a8725d9b 2023-03-30 16:56:04,784 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/21efb624d3c6404e9c90c1573c4a068f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/21efb624d3c6404e9c90c1573c4a068f 2023-03-30 16:56:04,785 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/3b3c1a893dcd4ae699382b4dc086585d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/3b3c1a893dcd4ae699382b4dc086585d 2023-03-30 16:56:04,786 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/4deecc1daac54ceeab1017db7545242b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/B/4deecc1daac54ceeab1017db7545242b 2023-03-30 16:56:04,788 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/5525119da53d4fc8ba7b44db2af4d584 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/5525119da53d4fc8ba7b44db2af4d584 2023-03-30 16:56:04,789 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/b7554de675204ede88981dc9e27d75d9 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/C/b7554de675204ede88981dc9e27d75d9 2023-03-30 16:56:04,791 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/recovered.edits/581.seqid to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79/recovered.edits/581.seqid 2023-03-30 16:56:04,791 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/4038776cda6db2eaa29896eab9262a79 2023-03-30 16:56:04,791 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 16:56:04,793 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:56:04,794 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-30 16:56:04,796 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-30 16:56:04,797 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:56:04,797 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-30 16:56:04,797 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680195364797"}]},"ts":"9223372036854775807"} 2023-03-30 16:56:04,798 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-30 16:56:04,798 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 4038776cda6db2eaa29896eab9262a79, NAME => 'TestAcidGuarantees,,1680195337393.4038776cda6db2eaa29896eab9262a79.', STARTKEY => '', ENDKEY => ''}] 2023-03-30 16:56:04,798 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-30 16:56:04,799 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680195364798"}]},"ts":"9223372036854775807"} 2023-03-30 16:56:04,800 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-30 16:56:04,802 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:56:04,802 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 32 msec 2023-03-30 16:56:04,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-30 16:56:04,874 INFO [Listener at localhost/38075] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-03-30 16:56:04,885 INFO [Listener at localhost/38075] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227 (was 229), OpenFileDescriptor=546 (was 517) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=415 (was 365) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=4747 (was 4351) - AvailableMemoryMB LEAK? - 2023-03-30 16:56:04,897 INFO [Listener at localhost/38075] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=227, OpenFileDescriptor=546, MaxFileDescriptor=60000, SystemLoadAverage=415, ProcessCount=169, AvailableMemoryMB=4735 2023-03-30 16:56:04,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-30 16:56:04,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-30 16:56:04,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-30 16:56:04,902 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-30 16:56:04,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-03-30 16:56:04,903 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-30 16:56:04,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-30 16:56:04,904 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:04,905 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487 empty. 2023-03-30 16:56:04,905 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:04,905 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 16:56:05,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-30 16:56:05,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-30 16:56:05,327 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-30 16:56:05,328 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 61fee4b7e4fa8ab90f65e22421bf6487, NAME => 'TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp 2023-03-30 16:56:05,333 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:56:05,333 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 61fee4b7e4fa8ab90f65e22421bf6487, disabling compactions & flushes 2023-03-30 16:56:05,333 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:05,333 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:05,333 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. after waiting 0 ms 2023-03-30 16:56:05,333 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:05,333 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:05,333 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:05,335 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-30 16:56:05,336 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195365336"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680195365336"}]},"ts":"1680195365336"} 2023-03-30 16:56:05,337 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-30 16:56:05,338 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-30 16:56:05,338 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195365338"}]},"ts":"1680195365338"} 2023-03-30 16:56:05,338 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-30 16:56:05,341 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=61fee4b7e4fa8ab90f65e22421bf6487, ASSIGN}] 2023-03-30 16:56:05,343 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=61fee4b7e4fa8ab90f65e22421bf6487, ASSIGN 2023-03-30 16:56:05,343 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=61fee4b7e4fa8ab90f65e22421bf6487, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,45193,1680195214285; forceNewPlan=false, retain=false 2023-03-30 16:56:05,494 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=61fee4b7e4fa8ab90f65e22421bf6487, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:05,494 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195365494"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680195365494"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680195365494"}]},"ts":"1680195365494"} 2023-03-30 16:56:05,496 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285}] 2023-03-30 16:56:05,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-30 16:56:05,652 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:05,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 61fee4b7e4fa8ab90f65e22421bf6487, NAME => 'TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.', STARTKEY => '', ENDKEY => ''} 2023-03-30 16:56:05,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:05,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:56:05,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:05,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:05,654 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:05,656 DEBUG [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A 2023-03-30 16:56:05,656 DEBUG [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A 2023-03-30 16:56:05,656 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:56:05,656 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61fee4b7e4fa8ab90f65e22421bf6487 columnFamilyName A 2023-03-30 16:56:05,657 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] regionserver.HStore(310): Store=61fee4b7e4fa8ab90f65e22421bf6487/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:56:05,657 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:05,658 DEBUG [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B 2023-03-30 16:56:05,658 DEBUG [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B 2023-03-30 16:56:05,658 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:56:05,658 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61fee4b7e4fa8ab90f65e22421bf6487 columnFamilyName B 2023-03-30 16:56:05,659 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] regionserver.HStore(310): Store=61fee4b7e4fa8ab90f65e22421bf6487/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:56:05,659 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:05,660 DEBUG [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C 2023-03-30 16:56:05,660 DEBUG [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C 2023-03-30 16:56:05,660 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:56:05,661 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61fee4b7e4fa8ab90f65e22421bf6487 columnFamilyName C 2023-03-30 16:56:05,661 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] regionserver.HStore(310): Store=61fee4b7e4fa8ab90f65e22421bf6487/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:56:05,661 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:05,662 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:05,662 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:05,664 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 16:56:05,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:05,667 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-30 16:56:05,668 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 61fee4b7e4fa8ab90f65e22421bf6487; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74403022, jitterRate=0.10869142413139343}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 16:56:05,668 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:05,668 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., pid=56, masterSystemTime=1680195365647 2023-03-30 16:56:05,669 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:05,670 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:05,670 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=61fee4b7e4fa8ab90f65e22421bf6487, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:05,670 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195365670"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680195365670"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680195365670"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680195365670"}]},"ts":"1680195365670"} 2023-03-30 16:56:05,672 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-03-30 16:56:05,672 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 in 175 msec 2023-03-30 16:56:05,673 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-03-30 16:56:05,673 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=61fee4b7e4fa8ab90f65e22421bf6487, ASSIGN in 331 msec 2023-03-30 16:56:05,673 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-30 16:56:05,674 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195365674"}]},"ts":"1680195365674"} 2023-03-30 16:56:05,674 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-30 16:56:05,681 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-30 16:56:05,682 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 782 msec 2023-03-30 16:56:06,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-30 16:56:06,007 INFO [Listener at localhost/38075] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-03-30 16:56:06,007 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x45d9b525 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:56:06,012 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22a2de11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:56:06,013 DEBUG [Listener at localhost/38075] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:56:06,014 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51106, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:56:06,015 DEBUG [Listener at localhost/38075] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-30 16:56:06,018 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55398, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-30 16:56:06,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-30 16:56:06,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-30 16:56:06,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-30 16:56:06,028 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-30 16:56:06,029 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-30 16:56:06,029 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-30 16:56:06,031 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-30 16:56:06,033 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=61fee4b7e4fa8ab90f65e22421bf6487, REOPEN/MOVE}] 2023-03-30 16:56:06,033 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=61fee4b7e4fa8ab90f65e22421bf6487, REOPEN/MOVE 2023-03-30 16:56:06,034 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=61fee4b7e4fa8ab90f65e22421bf6487, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,034 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195366034"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680195366034"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680195366034"}]},"ts":"1680195366034"} 2023-03-30 16:56:06,035 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285}] 2023-03-30 16:56:06,186 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:06,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 61fee4b7e4fa8ab90f65e22421bf6487, disabling compactions & flushes 2023-03-30 16:56:06,187 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:06,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:06,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. after waiting 0 ms 2023-03-30 16:56:06,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:06,190 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-30 16:56:06,191 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:06,191 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:06,191 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 61fee4b7e4fa8ab90f65e22421bf6487 to self. 2023-03-30 16:56:06,192 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:06,192 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=61fee4b7e4fa8ab90f65e22421bf6487, regionState=CLOSED 2023-03-30 16:56:06,192 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195366192"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680195366192"}]},"ts":"1680195366192"} 2023-03-30 16:56:06,194 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-03-30 16:56:06,194 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 in 158 msec 2023-03-30 16:56:06,195 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=61fee4b7e4fa8ab90f65e22421bf6487, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,45193,1680195214285; forceNewPlan=false, retain=true 2023-03-30 16:56:06,345 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=61fee4b7e4fa8ab90f65e22421bf6487, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,345 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195366345"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680195366345"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680195366345"}]},"ts":"1680195366345"} 2023-03-30 16:56:06,346 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285}] 2023-03-30 16:56:06,500 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:06,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 61fee4b7e4fa8ab90f65e22421bf6487, NAME => 'TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.', STARTKEY => '', ENDKEY => ''} 2023-03-30 16:56:06,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:06,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-30 16:56:06,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:06,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:06,502 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:06,503 DEBUG [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A 2023-03-30 16:56:06,503 DEBUG [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A 2023-03-30 16:56:06,503 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:56:06,503 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61fee4b7e4fa8ab90f65e22421bf6487 columnFamilyName A 2023-03-30 16:56:06,504 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] regionserver.HStore(310): Store=61fee4b7e4fa8ab90f65e22421bf6487/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:56:06,504 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:06,505 DEBUG [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B 2023-03-30 16:56:06,505 DEBUG [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B 2023-03-30 16:56:06,505 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:56:06,505 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61fee4b7e4fa8ab90f65e22421bf6487 columnFamilyName B 2023-03-30 16:56:06,506 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] regionserver.HStore(310): Store=61fee4b7e4fa8ab90f65e22421bf6487/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:56:06,506 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:06,506 DEBUG [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C 2023-03-30 16:56:06,506 DEBUG [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C 2023-03-30 16:56:06,506 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-30 16:56:06,507 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61fee4b7e4fa8ab90f65e22421bf6487 columnFamilyName C 2023-03-30 16:56:06,507 INFO [StoreOpener-61fee4b7e4fa8ab90f65e22421bf6487-1] regionserver.HStore(310): Store=61fee4b7e4fa8ab90f65e22421bf6487/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-30 16:56:06,507 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:06,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:06,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:06,510 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-30 16:56:06,511 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:06,512 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 61fee4b7e4fa8ab90f65e22421bf6487; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66598035, jitterRate=-0.007611945271492004}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-30 16:56:06,512 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:06,512 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., pid=61, masterSystemTime=1680195366498 2023-03-30 16:56:06,514 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:06,514 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:06,514 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=61fee4b7e4fa8ab90f65e22421bf6487, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,514 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195366514"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680195366514"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680195366514"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680195366514"}]},"ts":"1680195366514"} 2023-03-30 16:56:06,516 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-03-30 16:56:06,516 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 in 169 msec 2023-03-30 16:56:06,517 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-03-30 16:56:06,517 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=61fee4b7e4fa8ab90f65e22421bf6487, REOPEN/MOVE in 483 msec 2023-03-30 16:56:06,519 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-03-30 16:56:06,519 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 487 msec 2023-03-30 16:56:06,520 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 500 msec 2023-03-30 16:56:06,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-03-30 16:56:06,522 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x54cb8b37 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:56:06,528 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5cb37c16, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:56:06,529 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x041dc777 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:56:06,532 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@c0766f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:56:06,533 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x089b6154 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:56:06,536 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6257294f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:56:06,536 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x01d207bb to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:56:06,539 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52104714, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:56:06,539 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x5bcc330a to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:56:06,544 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a320817, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:56:06,545 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x19c1e3ee to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:56:06,551 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43693271, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:56:06,551 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x526f2221 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:56:06,556 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44277511, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:56:06,557 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x17cf2e3f to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:56:06,565 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c1a68cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:56:06,566 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x3420e54c to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:56:06,570 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a709b21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:56:06,570 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(139): Connect 0x26d7d0b5 to 127.0.0.1:58435 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-30 16:56:06,574 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@156e4ba6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-30 16:56:06,589 DEBUG [hconnection-0x1dd992a4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:56:06,589 DEBUG [hconnection-0x2b7f3101-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:56:06,589 DEBUG [hconnection-0x599f4c08-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:56:06,589 DEBUG [hconnection-0x23b8df68-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:56:06,590 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51118, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:56:06,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:06,591 DEBUG [hconnection-0x6c6d30dd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:56:06,592 DEBUG [hconnection-0x567ff737-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:56:06,592 DEBUG [hconnection-0x772c2fb9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:56:06,593 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51156, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:56:06,598 DEBUG [hconnection-0x1082853b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:56:06,599 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51116, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:56:06,599 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51140, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:56:06,600 DEBUG [hconnection-0xc024bd3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:56:06,601 DEBUG [hconnection-0x2f0f22a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-30 16:56:06,602 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51108, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:56:06,603 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51126, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:56:06,603 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51160, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:56:06,605 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51186, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:56:06,605 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51170, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:56:06,609 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51202, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-30 16:56:06,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:06,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:06,611 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:06,611 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:06,611 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:06,611 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:06,611 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,615 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:06,616 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:06,616 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:06,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:06,617 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:06,617 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:06,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:06,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:06,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:06,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:06,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:06,618 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:06,618 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:06,619 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:06,619 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:06,619 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:06,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:06,619 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:56:06,620 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:06,621 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:06,621 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:06,621 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:06,621 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:06,621 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:06,621 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:06,621 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:06,621 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:06,621 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,621 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:06,621 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:06,621 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:06,621 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:06,621 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:06,623 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:06,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:06,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:06,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:06,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:06,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,625 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,625 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,625 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:06,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:06,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:06,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72b1a35d[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:06,626 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:06,626 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,629 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,629 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:06,629 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:06,629 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:06,629 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,629 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:06,631 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:06,631 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:06,631 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:06,631 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:06,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:06,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:06,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:06,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:06,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:06,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:06,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:06,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:06,633 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:06,634 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:06,634 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:06,634 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:06,634 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:06,636 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,636 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:06,636 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:06,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:06,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:06,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:06,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:06,636 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:06,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:06,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195426634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:06,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195426635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680195366633, max timestamp=1680195366636Num uniques 859; 2023-03-30 16:56:06,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:56:06,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680195366633, max timestamp=1680195366636Num uniques 859; 2023-03-30 16:56:06,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680195366633, max timestamp=1680195366636Num uniques 859; 2023-03-30 16:56:06,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:06,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:56:06,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:56:06,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:06,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,640 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:06,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:06,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,643 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,643 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:06,643 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:06,643 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:06,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:06,643 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:06,644 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:06,644 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:06,643 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:06,643 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,644 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:06,644 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:06,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:06,644 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:06,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:06,663 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:06,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51170 deadline: 1680195426659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:06,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51126 deadline: 1680195426659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,663 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:06,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51108 deadline: 1680195426660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:06,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195426739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:06,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195426740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,744 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:06,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:06,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:06,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:06,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:06,752 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:06,752 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:06,752 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:06,752 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:06,755 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,755 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:06,758 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:06,758 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:06,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:06,758 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,758 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:06,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:06,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:06,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:06,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:06,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:06,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:06,759 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:06,759 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:06,759 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:06,759 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:06,759 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:06,759 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:06,759 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:06,760 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:06,760 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:06,760 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:06,760 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:06,760 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,760 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:06,760 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:06,761 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:06,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:06,761 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:06,761 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,762 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:06,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:06,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:06,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:06,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:06,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e809cd1[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:06,763 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:06,763 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,764 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,764 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:06,764 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:06,764 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:06,764 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,764 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:06,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:06,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51126 deadline: 1680195426764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:06,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51108 deadline: 1680195426765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:06,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51170 deadline: 1680195426765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,774 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,774 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:06,774 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:06,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:06,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:06,774 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:06,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:06,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:06,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:06,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:06,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:06,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:06,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:06,776 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:06,776 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:06,776 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:06,776 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:06,776 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:06,778 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,778 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,778 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:06,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:06,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:06,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:06,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:06,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:06,779 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:06,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:06,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,780 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:06,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,782 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,782 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:06,782 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:06,782 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,782 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,782 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,782 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,782 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:06,782 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:06,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:06,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:06,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:06,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:06,782 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:06,783 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:06,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:06,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:06,783 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:06,883 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:06,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:06,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:06,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:06,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:06,886 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:06,886 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:06,886 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:06,886 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:06,886 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,886 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:06,887 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,887 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:06,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:06,888 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,888 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:06,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:06,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:06,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:06,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:06,889 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:06,889 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:06,889 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:06,889 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:06,889 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:06,889 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:06,889 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:06,889 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:06,889 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:06,889 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:06,889 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:06,889 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,889 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:06,889 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:06,891 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:06,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:06,892 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:06,892 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,892 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:06,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:06,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:06,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:06,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:06,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3087b2f7[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:06,893 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:06,893 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,894 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,894 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:06,894 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:06,894 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:06,894 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,894 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:06,896 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,896 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:06,896 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:06,896 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:06,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:06,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:06,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:06,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:06,898 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:06,898 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:06,899 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:06,899 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:06,899 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:06,900 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,900 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:06,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:06,900 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:06,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:06,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:06,901 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:06,902 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:06,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:06,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,905 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:06,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:06,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:06,905 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:06,905 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,905 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:06,905 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:06,906 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:06,906 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,906 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,906 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:06,906 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:06,906 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:06,906 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:06,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:06,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:06,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:06,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:06,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:06,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195426943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:06,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195426943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:06,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51126 deadline: 1680195426968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:06,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51108 deadline: 1680195426969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:06,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:06,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51170 deadline: 1680195426969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,006 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:07,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:07,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:07,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:07,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:07,009 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:07,009 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:07,009 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:07,010 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:07,010 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:07,010 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:07,011 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,011 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:07,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:07,012 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:07,012 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:07,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:07,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:07,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:07,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:07,013 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:07,013 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:07,013 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:07,013 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:07,013 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:07,013 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:07,013 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:07,013 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:07,013 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:07,014 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:07,014 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:07,014 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,014 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:07,014 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:07,016 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:07,016 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:07,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:07,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:07,017 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:07,017 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:07,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:07,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ec99420[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:07,018 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:07,018 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:07,019 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:07,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:07,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:07,019 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:07,019 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:07,019 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:07,019 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,019 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:07,022 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,022 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:07,022 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:07,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:07,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:07,022 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:07,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:07,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:07,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:07,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:07,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:07,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:07,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:07,024 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:07,024 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:07,024 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:07,024 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:07,024 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:07,026 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:07,026 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:07,027 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:07,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:07,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:07,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:07,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:07,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:07,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:07,027 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:07,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:07,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:07,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,028 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:07,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:07,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,031 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:07,031 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:07,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:07,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:07,031 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,031 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:07,031 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:07,031 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,031 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:07,031 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:07,031 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:07,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:07,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:07,031 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:07,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:07,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:07,032 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:07,032 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:07,048 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303305c7dbc2639a1442b890fdeb30bf0b27f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303305c7dbc2639a1442b890fdeb30bf0b27f 2023-03-30 16:56:07,061 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/8b1424aef8bf4700b26de1257ea6151b 2023-03-30 16:56:07,082 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/e0406609c4e44f5a82c5c394b8b131b5 2023-03-30 16:56:07,102 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/8eab36bff36e402b9599d72184d474cb 2023-03-30 16:56:07,106 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/8b1424aef8bf4700b26de1257ea6151b as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/8b1424aef8bf4700b26de1257ea6151b 2023-03-30 16:56:07,110 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/8b1424aef8bf4700b26de1257ea6151b, entries=150, sequenceid=15, filesize=25.1 K 2023-03-30 16:56:07,111 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/e0406609c4e44f5a82c5c394b8b131b5 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e0406609c4e44f5a82c5c394b8b131b5 2023-03-30 16:56:07,115 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e0406609c4e44f5a82c5c394b8b131b5, entries=150, sequenceid=15, filesize=11.6 K 2023-03-30 16:56:07,116 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/8eab36bff36e402b9599d72184d474cb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/8eab36bff36e402b9599d72184d474cb 2023-03-30 16:56:07,119 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/8eab36bff36e402b9599d72184d474cb, entries=150, sequenceid=15, filesize=11.6 K 2023-03-30 16:56:07,120 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 61fee4b7e4fa8ab90f65e22421bf6487 in 501ms, sequenceid=15, compaction requested=false 2023-03-30 16:56:07,120 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:07,132 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:07,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:07,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:07,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:07,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:07,135 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:07,135 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:07,135 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:07,135 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:07,135 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:07,135 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:07,137 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:07,137 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:07,137 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,137 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:07,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:07,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:07,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:07,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:07,138 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:07,138 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:07,138 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:07,138 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:07,138 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:07,139 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:07,139 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:07,139 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-30 16:56:07,139 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:07,139 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:07,139 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:07,139 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:07,140 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:07,140 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:07,161 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330552783560fbe40409f1ee0663eba6e6d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330552783560fbe40409f1ee0663eba6e6d 2023-03-30 16:56:07,167 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/f1e526a61ed545f194c355dcffae768e 2023-03-30 16:56:07,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:07,248 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:07,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:07,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195427258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:07,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195427260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:07,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51126 deadline: 1680195427270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:07,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51108 deadline: 1680195427273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:07,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51170 deadline: 1680195427273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:07,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195427361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:07,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195427363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:07,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195427564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:07,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195427567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,577 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/351b383f591a4c05892f49d01db57783 2023-03-30 16:56:07,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:07,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51126 deadline: 1680195427773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:07,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51108 deadline: 1680195427775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:07,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51170 deadline: 1680195427777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:07,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195427869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:07,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195427873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:07,987 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/b116ad30f0dc42a9ab5345884486380d 2023-03-30 16:56:07,991 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/f1e526a61ed545f194c355dcffae768e as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/f1e526a61ed545f194c355dcffae768e 2023-03-30 16:56:07,994 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/f1e526a61ed545f194c355dcffae768e, entries=150, sequenceid=50, filesize=25.1 K 2023-03-30 16:56:07,995 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/351b383f591a4c05892f49d01db57783 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/351b383f591a4c05892f49d01db57783 2023-03-30 16:56:07,999 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/351b383f591a4c05892f49d01db57783, entries=150, sequenceid=50, filesize=11.6 K 2023-03-30 16:56:08,000 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/b116ad30f0dc42a9ab5345884486380d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/b116ad30f0dc42a9ab5345884486380d 2023-03-30 16:56:08,003 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/b116ad30f0dc42a9ab5345884486380d, entries=150, sequenceid=50, filesize=11.6 K 2023-03-30 16:56:08,003 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 61fee4b7e4fa8ab90f65e22421bf6487 in 864ms, sequenceid=50, compaction requested=false 2023-03-30 16:56:08,004 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:08,004 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:08,004 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:08,004 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:08,004 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,004 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:08,004 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:08,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,007 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:08,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:08,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@419d63ef[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:08,009 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:08,009 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,011 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,011 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:08,011 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:08,011 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:08,011 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,011 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:08,013 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:08,013 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:08,013 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:08,013 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:08,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:08,016 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:08,016 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:08,016 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:08,016 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:08,016 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:08,021 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,021 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,021 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,022 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:08,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:08,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,023 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,026 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:08,026 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:08,026 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:08,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:08,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:08,026 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:08,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:08,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:08,026 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,026 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:08,026 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,026 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:08,027 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,027 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,027 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:08,027 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,127 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:08,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:08,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:08,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:08,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:08,130 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:08,130 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:08,131 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:08,131 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:08,131 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,131 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,132 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:08,132 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:08,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:08,133 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,133 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:08,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:08,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:08,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:08,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:08,134 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:08,134 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:08,134 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:08,134 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:08,134 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:08,134 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:08,135 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:08,135 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 16:56:08,135 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:08,135 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:08,135 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:08,135 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:08,135 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:08,135 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:08,164 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330d81d5f468f774fa482b9c98b88559e1d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d81d5f468f774fa482b9c98b88559e1d 2023-03-30 16:56:08,169 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=61, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/4eacc451ff45415494229c9c2877d4a6 2023-03-30 16:56:08,183 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/44fc1250f04f4bd98c67248f9da50942 2023-03-30 16:56:08,194 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/2dd16d7e958c4bb3b5119c22cff0130c 2023-03-30 16:56:08,198 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/4eacc451ff45415494229c9c2877d4a6 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/4eacc451ff45415494229c9c2877d4a6 2023-03-30 16:56:08,202 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/4eacc451ff45415494229c9c2877d4a6, entries=150, sequenceid=61, filesize=25.1 K 2023-03-30 16:56:08,203 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/44fc1250f04f4bd98c67248f9da50942 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/44fc1250f04f4bd98c67248f9da50942 2023-03-30 16:56:08,218 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/44fc1250f04f4bd98c67248f9da50942, entries=150, sequenceid=61, filesize=11.6 K 2023-03-30 16:56:08,219 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/2dd16d7e958c4bb3b5119c22cff0130c as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/2dd16d7e958c4bb3b5119c22cff0130c 2023-03-30 16:56:08,223 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/2dd16d7e958c4bb3b5119c22cff0130c, entries=150, sequenceid=61, filesize=11.6 K 2023-03-30 16:56:08,223 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 61fee4b7e4fa8ab90f65e22421bf6487 in 88ms, sequenceid=61, compaction requested=true 2023-03-30 16:56:08,224 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:08,224 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:08,224 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:08,224 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:08,224 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,224 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:08,224 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:08,225 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,225 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,226 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,226 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:08,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:08,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b111919[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:08,228 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:08,228 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,230 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,230 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:08,230 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:08,230 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:08,230 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,230 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:08,231 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,231 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:08,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,232 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:08,232 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:08,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:08,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:08,234 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:08,234 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:08,234 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:08,234 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:08,234 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:08,236 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,236 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:08,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:08,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:08,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,241 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,242 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,249 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,252 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:08,252 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:08,252 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:08,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:08,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:08,252 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:08,252 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,252 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:08,253 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:08,253 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:08,253 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,253 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,253 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,253 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:08,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:08,353 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:08,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:08,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:08,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:08,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:08,357 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:08,357 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:08,357 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:08,357 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:08,357 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,357 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,359 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:08,359 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:08,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:08,360 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,360 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:08,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:08,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:08,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:08,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:08,361 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:08,361 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:08,361 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:08,361 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:08,361 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:08,361 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:08,361 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:08,362 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:08,362 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:08,362 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:08,362 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:08,362 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,362 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:08,362 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:08,363 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,363 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,364 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,364 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:08,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:08,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@260c1686[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:08,366 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:08,366 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,368 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,368 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,368 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:08,368 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:08,368 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:08,368 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:08,369 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,370 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:08,370 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:08,370 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:08,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:08,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:08,372 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:08,372 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:08,372 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:08,372 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:08,373 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:08,376 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,376 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:08,376 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,376 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:08,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:08,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,378 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,380 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:08,380 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:08,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:08,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:08,381 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,381 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,381 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,381 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,381 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,381 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,381 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:08,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:08,381 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:08,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:08,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:08,381 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:08,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:08,381 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:08,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:08,438 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 16:56:08,438 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:08,439 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:08,439 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:08,439 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:08,439 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:08,439 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:08,476 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303308b9d18663af342ff93adb2761e0abc94 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303308b9d18663af342ff93adb2761e0abc94 2023-03-30 16:56:08,482 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:08,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:08,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:08,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195368483, max timestamp=1680195368492Num uniques 859; 2023-03-30 16:56:08,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:56:08,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195368483, max timestamp=1680195368492Num uniques 859; 2023-03-30 16:56:08,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195368483, max timestamp=1680195368492Num uniques 859; 2023-03-30 16:56:08,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:56:08,496 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:56:08,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:08,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:08,500 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:08,500 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:08,500 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:08,500 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:08,500 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=73, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/72f2ff5fa4384a449089a2008ca208eb 2023-03-30 16:56:08,501 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,501 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,502 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:08,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:08,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:08,503 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,503 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,503 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:08,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:08,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:08,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:08,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:08,505 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:08,505 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:08,505 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:08,516 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:08,516 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:08,519 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:08,519 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:08,519 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:08,519 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:08,519 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:08,519 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:08,520 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,520 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:08,520 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:08,521 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,522 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,522 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:08,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5496fef3[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:08,523 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:08,523 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,526 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,526 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:08,526 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:08,527 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:08,527 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,527 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:08,528 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:08,528 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:08,528 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:08,528 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:08,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:08,531 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:08,531 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:08,531 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:08,531 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:08,531 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:08,532 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,532 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,532 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:08,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:08,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:08,533 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,535 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,536 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=73 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/9882a5fed8554c5db253ff27f9f3cdf3 2023-03-30 16:56:08,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,542 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:08,542 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:08,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:08,542 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:08,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:08,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:08,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:08,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:08,542 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,543 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,543 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,543 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,543 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:08,543 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:08,543 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,543 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,543 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:08,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:08,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:08,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195428546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:08,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195428548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,565 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=73 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/f4f2dc3e955449109da3549eacdaa0eb 2023-03-30 16:56:08,569 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/72f2ff5fa4384a449089a2008ca208eb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/72f2ff5fa4384a449089a2008ca208eb 2023-03-30 16:56:08,573 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/72f2ff5fa4384a449089a2008ca208eb, entries=350, sequenceid=73, filesize=52.3 K 2023-03-30 16:56:08,574 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/9882a5fed8554c5db253ff27f9f3cdf3 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/9882a5fed8554c5db253ff27f9f3cdf3 2023-03-30 16:56:08,578 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/9882a5fed8554c5db253ff27f9f3cdf3, entries=150, sequenceid=73, filesize=11.6 K 2023-03-30 16:56:08,579 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/f4f2dc3e955449109da3549eacdaa0eb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/f4f2dc3e955449109da3549eacdaa0eb 2023-03-30 16:56:08,583 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/f4f2dc3e955449109da3549eacdaa0eb, entries=150, sequenceid=73, filesize=11.6 K 2023-03-30 16:56:08,583 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 61fee4b7e4fa8ab90f65e22421bf6487 in 145ms, sequenceid=73, compaction requested=true 2023-03-30 16:56:08,583 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:08,583 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:56:08,583 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:56:08,583 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:56:08,584 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:56:08,584 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:56:08,585 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 130748 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:56:08,585 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 61fee4b7e4fa8ab90f65e22421bf6487/A is initiating minor compaction (all files) 2023-03-30 16:56:08,585 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:56:08,585 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61fee4b7e4fa8ab90f65e22421bf6487/A in TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:08,585 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 61fee4b7e4fa8ab90f65e22421bf6487/B is initiating minor compaction (all files) 2023-03-30 16:56:08,585 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/8b1424aef8bf4700b26de1257ea6151b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/f1e526a61ed545f194c355dcffae768e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/4eacc451ff45415494229c9c2877d4a6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/72f2ff5fa4384a449089a2008ca208eb] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp, totalSize=127.7 K 2023-03-30 16:56:08,585 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61fee4b7e4fa8ab90f65e22421bf6487/B in TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:08,585 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e0406609c4e44f5a82c5c394b8b131b5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/351b383f591a4c05892f49d01db57783, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/44fc1250f04f4bd98c67248f9da50942, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/9882a5fed8554c5db253ff27f9f3cdf3] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp, totalSize=46.4 K 2023-03-30 16:56:08,586 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 8b1424aef8bf4700b26de1257ea6151b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680195366617 2023-03-30 16:56:08,586 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting e0406609c4e44f5a82c5c394b8b131b5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680195366617 2023-03-30 16:56:08,586 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 351b383f591a4c05892f49d01db57783, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680195366643 2023-03-30 16:56:08,586 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting f1e526a61ed545f194c355dcffae768e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680195366643 2023-03-30 16:56:08,587 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 44fc1250f04f4bd98c67248f9da50942, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680195367249 2023-03-30 16:56:08,588 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 4eacc451ff45415494229c9c2877d4a6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680195367249 2023-03-30 16:56:08,588 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 9882a5fed8554c5db253ff27f9f3cdf3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1680195368432 2023-03-30 16:56:08,588 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 72f2ff5fa4384a449089a2008ca208eb, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1680195368394 2023-03-30 16:56:08,614 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 61fee4b7e4fa8ab90f65e22421bf6487#B#compaction#470 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:56:08,643 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:08,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:08,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:08,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:08,657 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-30 16:56:08,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680195368541, max timestamp=1680195368656Num uniques 1018; 2023-03-30 16:56:08,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-30 16:56:08,659 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:08,659 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:08,659 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:08,659 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-30 16:56:08,659 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:08,659 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-30 16:56:08,670 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61fee4b7e4fa8ab90f65e22421bf6487#A#compaction#471 average throughput is 0.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:56:08,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:08,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:08,672 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:08,672 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:08,672 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:08,672 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:08,672 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,673 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,674 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:08,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:08,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:08,674 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,674 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,674 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:08,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:08,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:08,678 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/d45291ad441e4cfc8769557dc30c8c42 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/d45291ad441e4cfc8769557dc30c8c42 2023-03-30 16:56:08,689 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61fee4b7e4fa8ab90f65e22421bf6487/B of 61fee4b7e4fa8ab90f65e22421bf6487 into d45291ad441e4cfc8769557dc30c8c42(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:56:08,689 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:08,689 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., storeName=61fee4b7e4fa8ab90f65e22421bf6487/B, priority=12, startTime=1680195368583; duration=0sec 2023-03-30 16:56:08,689 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:56:08,689 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:56:08,690 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:56:08,690 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 61fee4b7e4fa8ab90f65e22421bf6487/C is initiating minor compaction (all files) 2023-03-30 16:56:08,690 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61fee4b7e4fa8ab90f65e22421bf6487/C in TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:08,690 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/8eab36bff36e402b9599d72184d474cb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/b116ad30f0dc42a9ab5345884486380d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/2dd16d7e958c4bb3b5119c22cff0130c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/f4f2dc3e955449109da3549eacdaa0eb] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp, totalSize=46.4 K 2023-03-30 16:56:08,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:08,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:08,691 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:08,691 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:08,691 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:08,691 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 8eab36bff36e402b9599d72184d474cb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680195366617 2023-03-30 16:56:08,692 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting b116ad30f0dc42a9ab5345884486380d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680195366643 2023-03-30 16:56:08,692 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 2dd16d7e958c4bb3b5119c22cff0130c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680195367249 2023-03-30 16:56:08,693 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting f4f2dc3e955449109da3549eacdaa0eb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1680195368432 2023-03-30 16:56:08,696 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:08,696 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:08,696 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:08,696 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:08,696 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:08,696 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:08,696 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:08,696 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:08,696 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,701 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:08,701 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:08,702 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 61fee4b7e4fa8ab90f65e22421bf6487#C#compaction#473 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:56:08,702 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,702 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,703 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,703 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:08,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:08,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ebe6d95[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:08,704 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:08,704 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,705 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:08,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:08,705 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,705 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:08,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:08,708 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,708 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:08,708 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,708 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:08,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:08,710 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:08,710 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:08,710 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:08,710 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:08,710 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:08,711 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,711 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,711 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:08,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,713 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,714 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,716 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:08,716 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:08,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:08,717 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,717 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,717 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,717 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:08,717 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:08,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:08,717 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:08,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:08,717 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,717 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:08,717 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:08,717 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:08,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195428692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:08,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195428693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:08,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51170 deadline: 1680195428781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:08,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51126 deadline: 1680195428784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:08,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51108 deadline: 1680195428785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,798 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303305c7627fc76164ed48516e440929ca421 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303305c7627fc76164ed48516e440929ca421 2023-03-30 16:56:08,810 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/27a663c5e1d24dc68d27197f498c0015 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/27a663c5e1d24dc68d27197f498c0015 2023-03-30 16:56:08,815 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61fee4b7e4fa8ab90f65e22421bf6487/C of 61fee4b7e4fa8ab90f65e22421bf6487 into 27a663c5e1d24dc68d27197f498c0015(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:56:08,815 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:08,815 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., storeName=61fee4b7e4fa8ab90f65e22421bf6487/C, priority=12, startTime=1680195368584; duration=0sec 2023-03-30 16:56:08,815 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:56:08,817 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:08,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:08,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:08,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:08,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:08,824 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:08,824 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:08,824 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:08,824 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:08,825 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,825 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,826 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:08,826 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,826 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,826 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:08,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:08,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:08,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:08,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:08,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:08,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195428820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:08,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195428820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:08,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:08,834 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:08,835 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:08,835 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:08,840 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:08,840 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:08,840 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:08,840 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:08,840 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:08,840 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:08,840 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:08,840 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:08,840 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,840 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:08,840 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:08,843 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:08,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d447a8a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:08,845 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:08,845 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,847 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,847 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:08,847 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:08,847 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:08,847 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,847 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:08,849 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:08,849 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:08,849 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:08,849 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:08,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:08,852 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:08,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:08,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:08,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:08,852 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:08,854 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,854 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:08,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,854 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:08,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:08,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,856 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,858 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,862 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:08,862 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:08,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:08,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:08,863 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,863 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,863 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,863 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:08,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:08,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:08,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:08,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:08,863 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,863 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,863 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,863 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:08,863 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:08,864 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:08,964 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:08,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:08,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:08,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:08,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:08,968 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:08,968 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:08,968 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:08,968 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:08,968 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,968 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,969 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,969 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:08,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:08,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:08,970 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,970 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:08,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:08,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:08,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:08,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:08,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:08,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:08,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:08,972 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:08,972 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:08,972 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:08,972 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:08,972 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:08,972 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:08,972 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:08,972 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:08,972 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,972 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:08,972 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:08,974 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,974 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,975 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,975 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:08,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:08,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b3aff82[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:08,977 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:08,977 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,978 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,979 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:08,979 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:08,979 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:08,979 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,979 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:08,981 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,981 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:08,981 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:08,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,981 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:08,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:08,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:08,985 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:08,985 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:08,985 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:08,985 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:08,985 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:08,987 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,987 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,987 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:08,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:08,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:08,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:08,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:08,988 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:08,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:08,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,990 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:08,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:08,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,996 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,996 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,996 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:08,996 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:08,996 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:08,996 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:08,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:08,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:08,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:08,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:08,997 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:08,997 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,997 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:08,997 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:08,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:08,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:08,997 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:08,997 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:08,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:09,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:09,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195429032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:09,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195429032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,098 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:09,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:09,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:09,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:09,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:09,106 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:09,106 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:09,106 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:09,106 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:09,107 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,107 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:09,109 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:09,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:09,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:09,109 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,110 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,110 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:09,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:09,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:09,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:09,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:09,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:09,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:09,112 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:09,112 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:09,112 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:09,112 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:09,112 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:09,112 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:09,112 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:09,113 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:09,113 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:09,113 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:09,113 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:09,113 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,113 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:09,113 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:09,115 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,115 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:09,115 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,115 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:09,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:09,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:09,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:09,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78b8fb70[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:09,117 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:09,117 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,118 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,118 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:09,118 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:09,118 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:09,118 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,118 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:09,120 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:09,120 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:09,120 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:09,120 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:09,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:09,124 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:09,124 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:09,124 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:09,124 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:09,124 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:09,126 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,126 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:09,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,127 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:09,127 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:09,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:09,129 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:09,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:09,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,131 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,131 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:09,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:09,131 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:09,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:09,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:09,131 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:09,132 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:09,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:09,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:09,132 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:09,132 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,132 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,132 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,132 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,132 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:09,132 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:09,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:09,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:09,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:09,230 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/0e63f8fe412743719ef82aa27c5a5cf3 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/0e63f8fe412743719ef82aa27c5a5cf3 2023-03-30 16:56:09,232 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:09,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:09,234 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=111, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/e2918dba924b44f1a3116ea326ff5055 2023-03-30 16:56:09,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:09,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:09,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:09,237 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:09,237 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:09,237 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:09,237 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:09,238 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61fee4b7e4fa8ab90f65e22421bf6487/A of 61fee4b7e4fa8ab90f65e22421bf6487 into 0e63f8fe412743719ef82aa27c5a5cf3(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:56:09,238 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:09,238 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., storeName=61fee4b7e4fa8ab90f65e22421bf6487/A, priority=12, startTime=1680195368583; duration=0sec 2023-03-30 16:56:09,238 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,238 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:09,238 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:56:09,240 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:09,240 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:09,240 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,240 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:09,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:09,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:09,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:09,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:09,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:09,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:09,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:09,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:09,260 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:09,260 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:09,260 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:09,260 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:09,260 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:09,260 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:09,260 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:09,260 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:09,260 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,260 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:09,260 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:09,263 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,264 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:09,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:09,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:09,265 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,265 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:09,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:09,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:09,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:09,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:09,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7567f337[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:09,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:09,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,270 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:09,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:09,270 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:09,270 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,270 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:09,271 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,272 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:09,272 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:09,272 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:09,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:09,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:09,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:09,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:09,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:09,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:09,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:09,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:09,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:09,275 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:09,275 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:09,275 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:09,275 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:09,275 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:09,277 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:09,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:09,277 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,277 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:09,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:09,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:09,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:09,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,278 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:09,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:09,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:09,280 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:09,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:09,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,283 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,283 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:09,283 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:09,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:09,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:09,283 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:09,283 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:09,283 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:09,283 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:09,283 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:09,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:09,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:09,283 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,284 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,284 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,284 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:09,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:09,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:09,298 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/d545b4acac9845b594504810efbfd36d 2023-03-30 16:56:09,331 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/d868df44cf53422f9ebb81c6c20976be 2023-03-30 16:56:09,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:09,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195429337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,340 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/e2918dba924b44f1a3116ea326ff5055 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/e2918dba924b44f1a3116ea326ff5055 2023-03-30 16:56:09,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:09,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195429339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,345 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/e2918dba924b44f1a3116ea326ff5055, entries=200, sequenceid=111, filesize=31.9 K 2023-03-30 16:56:09,345 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/d545b4acac9845b594504810efbfd36d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/d545b4acac9845b594504810efbfd36d 2023-03-30 16:56:09,350 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/d545b4acac9845b594504810efbfd36d, entries=150, sequenceid=111, filesize=11.6 K 2023-03-30 16:56:09,351 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/d868df44cf53422f9ebb81c6c20976be as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/d868df44cf53422f9ebb81c6c20976be 2023-03-30 16:56:09,355 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/d868df44cf53422f9ebb81c6c20976be, entries=150, sequenceid=111, filesize=11.6 K 2023-03-30 16:56:09,355 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~105.11 KB/107630, heapSize ~297.23 KB/304368, currentSize=73.80 KB/75570 for 61fee4b7e4fa8ab90f65e22421bf6487 in 698ms, sequenceid=111, compaction requested=false 2023-03-30 16:56:09,355 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:09,383 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:09,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:09,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:09,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:09,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:09,396 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:09,396 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:09,396 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:09,396 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:09,397 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,397 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:09,402 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,402 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:09,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:09,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:09,402 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,402 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:09,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:09,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:09,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:09,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:09,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:09,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:09,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:09,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:09,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:09,416 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:09,416 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:09,428 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:09,428 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:09,428 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-30 16:56:09,428 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:09,428 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:09,428 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:09,429 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:09,429 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:09,429 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:09,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:09,842 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:09,861 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330646009feb7bf48fab0755c810ff8b4af to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330646009feb7bf48fab0755c810ff8b4af 2023-03-30 16:56:09,867 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=128, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/06855a59aab54694b50d0e025d3eb66a 2023-03-30 16:56:09,871 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:09,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195429869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:09,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195429870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,881 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/565691543d9d45a4b5255e500d72c1c9 2023-03-30 16:56:09,913 WARN [DataStreamer for file /user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/30a37962eddc4037b9fef953b7c06132] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-03-30 16:56:09,916 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/30a37962eddc4037b9fef953b7c06132 2023-03-30 16:56:09,920 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/06855a59aab54694b50d0e025d3eb66a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/06855a59aab54694b50d0e025d3eb66a 2023-03-30 16:56:09,924 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/06855a59aab54694b50d0e025d3eb66a, entries=150, sequenceid=128, filesize=25.1 K 2023-03-30 16:56:09,924 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/565691543d9d45a4b5255e500d72c1c9 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/565691543d9d45a4b5255e500d72c1c9 2023-03-30 16:56:09,929 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/565691543d9d45a4b5255e500d72c1c9, entries=150, sequenceid=128, filesize=11.6 K 2023-03-30 16:56:09,930 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/30a37962eddc4037b9fef953b7c06132 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/30a37962eddc4037b9fef953b7c06132 2023-03-30 16:56:09,933 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/30a37962eddc4037b9fef953b7c06132, entries=150, sequenceid=128, filesize=11.6 K 2023-03-30 16:56:09,934 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 61fee4b7e4fa8ab90f65e22421bf6487 in 506ms, sequenceid=128, compaction requested=true 2023-03-30 16:56:09,934 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:09,934 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:09,934 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:09,934 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:09,934 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,934 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:09,934 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:09,936 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:09,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:09,936 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:09,936 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,936 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:09,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:09,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b09573b[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:09,938 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:09,938 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,941 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,941 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:09,941 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:09,941 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:09,942 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,942 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:09,943 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,943 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:09,943 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:09,943 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:09,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:09,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:09,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:09,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:09,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:09,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:09,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:09,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:09,946 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:09,946 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:09,946 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:09,946 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:09,946 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:09,949 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,949 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:09,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,949 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:09,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:09,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:09,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:09,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:09,950 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:09,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:09,952 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:09,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:09,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,954 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:09,954 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:09,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:09,954 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:09,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:09,954 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:09,954 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:09,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:09,954 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:09,954 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:09,955 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:09,955 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,955 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,955 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:09,955 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:09,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:09,974 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-30 16:56:09,975 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:09,975 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:09,975 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:09,975 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:09,975 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:09,975 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:09,996 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330558e741e6832493c9e9d8bf0be9f46bb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330558e741e6832493c9e9d8bf0be9f46bb 2023-03-30 16:56:09,999 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:09,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:09,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195429995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:09,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195429995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,003 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=148, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/3f82c4b9a84e4aeebdddaa5234b99bb2 2023-03-30 16:56:10,029 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/05a654eb9f6044a0baa1d7198eb97252 2023-03-30 16:56:10,046 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/347ccba285484b8b80c6499df9f0e7f7 2023-03-30 16:56:10,051 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/3f82c4b9a84e4aeebdddaa5234b99bb2 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/3f82c4b9a84e4aeebdddaa5234b99bb2 2023-03-30 16:56:10,054 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/3f82c4b9a84e4aeebdddaa5234b99bb2, entries=150, sequenceid=148, filesize=25.3 K 2023-03-30 16:56:10,055 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:10,055 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/05a654eb9f6044a0baa1d7198eb97252 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/05a654eb9f6044a0baa1d7198eb97252 2023-03-30 16:56:10,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:10,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:10,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:10,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:10,058 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:10,058 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:10,058 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:10,058 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:10,059 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,059 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,059 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/05a654eb9f6044a0baa1d7198eb97252, entries=150, sequenceid=148, filesize=11.8 K 2023-03-30 16:56:10,060 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/347ccba285484b8b80c6499df9f0e7f7 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/347ccba285484b8b80c6499df9f0e7f7 2023-03-30 16:56:10,061 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:10,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:10,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:10,061 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,061 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,061 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:10,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:10,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:10,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:10,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:10,063 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:10,063 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:10,063 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:10,064 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/347ccba285484b8b80c6499df9f0e7f7, entries=150, sequenceid=148, filesize=11.8 K 2023-03-30 16:56:10,064 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:10,064 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:10,064 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:10,064 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:10,064 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:10,064 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:10,064 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:10,064 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:10,064 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,064 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:10,064 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:10,064 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 61fee4b7e4fa8ab90f65e22421bf6487 in 90ms, sequenceid=148, compaction requested=true 2023-03-30 16:56:10,065 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:10,065 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:56:10,065 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:56:10,065 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:56:10,065 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:56:10,065 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:56:10,066 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47832 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:56:10,066 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 61fee4b7e4fa8ab90f65e22421bf6487/B is initiating minor compaction (all files) 2023-03-30 16:56:10,066 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61fee4b7e4fa8ab90f65e22421bf6487/B in TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:10,066 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110212 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:56:10,066 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/d45291ad441e4cfc8769557dc30c8c42, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/d545b4acac9845b594504810efbfd36d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/565691543d9d45a4b5255e500d72c1c9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/05a654eb9f6044a0baa1d7198eb97252] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp, totalSize=46.7 K 2023-03-30 16:56:10,066 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 61fee4b7e4fa8ab90f65e22421bf6487/A is initiating minor compaction (all files) 2023-03-30 16:56:10,066 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61fee4b7e4fa8ab90f65e22421bf6487/A in TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:10,066 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/0e63f8fe412743719ef82aa27c5a5cf3, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/e2918dba924b44f1a3116ea326ff5055, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/06855a59aab54694b50d0e025d3eb66a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/3f82c4b9a84e4aeebdddaa5234b99bb2] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp, totalSize=107.6 K 2023-03-30 16:56:10,067 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting d45291ad441e4cfc8769557dc30c8c42, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1680195368432 2023-03-30 16:56:10,067 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 0e63f8fe412743719ef82aa27c5a5cf3, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1680195368432 2023-03-30 16:56:10,067 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting d545b4acac9845b594504810efbfd36d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680195368541 2023-03-30 16:56:10,067 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting e2918dba924b44f1a3116ea326ff5055, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680195368541 2023-03-30 16:56:10,067 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 565691543d9d45a4b5255e500d72c1c9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1680195368685 2023-03-30 16:56:10,068 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 06855a59aab54694b50d0e025d3eb66a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1680195368685 2023-03-30 16:56:10,068 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 05a654eb9f6044a0baa1d7198eb97252, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1680195369868 2023-03-30 16:56:10,068 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 3f82c4b9a84e4aeebdddaa5234b99bb2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1680195369868 2023-03-30 16:56:10,070 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,070 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:10,071 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,071 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:10,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:10,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:10,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:10,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:10,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47037f37[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:10,073 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:10,073 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,074 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,075 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:10,075 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:10,075 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:10,075 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,075 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:10,076 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:10,076 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:10,076 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:10,076 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:10,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:10,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:10,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:10,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,080 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 61fee4b7e4fa8ab90f65e22421bf6487#B#compaction#482 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:56:10,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:10,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:10,082 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:10,082 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:10,082 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:10,082 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:10,082 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:10,084 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:10,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:10,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:10,085 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,085 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,085 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:10,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:10,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:10,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,087 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,092 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61fee4b7e4fa8ab90f65e22421bf6487#A#compaction#483 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:56:10,094 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,094 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,094 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,094 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,094 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,094 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,094 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,095 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:10,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:10,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:10,095 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:10,095 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:10,096 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/4cbafb90084a40b2a4d136b914653234 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/4cbafb90084a40b2a4d136b914653234 2023-03-30 16:56:10,097 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:10,097 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:10,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:10,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:10,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:10,101 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61fee4b7e4fa8ab90f65e22421bf6487/B of 61fee4b7e4fa8ab90f65e22421bf6487 into 4cbafb90084a40b2a4d136b914653234(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:56:10,102 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:10,102 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., storeName=61fee4b7e4fa8ab90f65e22421bf6487/B, priority=12, startTime=1680195370065; duration=0sec 2023-03-30 16:56:10,102 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:56:10,102 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:56:10,103 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47832 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:56:10,103 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 61fee4b7e4fa8ab90f65e22421bf6487/C is initiating minor compaction (all files) 2023-03-30 16:56:10,103 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61fee4b7e4fa8ab90f65e22421bf6487/C in TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:10,103 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/27a663c5e1d24dc68d27197f498c0015, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/d868df44cf53422f9ebb81c6c20976be, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/30a37962eddc4037b9fef953b7c06132, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/347ccba285484b8b80c6499df9f0e7f7] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp, totalSize=46.7 K 2023-03-30 16:56:10,103 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 27a663c5e1d24dc68d27197f498c0015, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1680195368432 2023-03-30 16:56:10,104 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting d868df44cf53422f9ebb81c6c20976be, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680195368541 2023-03-30 16:56:10,104 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 30a37962eddc4037b9fef953b7c06132, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1680195368685 2023-03-30 16:56:10,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:10,104 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-30 16:56:10,105 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 347ccba285484b8b80c6499df9f0e7f7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1680195369868 2023-03-30 16:56:10,105 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:10,106 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:10,106 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:10,106 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:10,106 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:10,106 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:10,114 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 61fee4b7e4fa8ab90f65e22421bf6487#C#compaction#485 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:56:10,131 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330e55bbfb02b95433a8955e438e2772992 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330e55bbfb02b95433a8955e438e2772992 2023-03-30 16:56:10,136 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/4f253933dbba40e9b915944427cadec3 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/4f253933dbba40e9b915944427cadec3 2023-03-30 16:56:10,141 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61fee4b7e4fa8ab90f65e22421bf6487/C of 61fee4b7e4fa8ab90f65e22421bf6487 into 4f253933dbba40e9b915944427cadec3(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:56:10,141 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:10,141 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., storeName=61fee4b7e4fa8ab90f65e22421bf6487/C, priority=12, startTime=1680195370065; duration=0sec 2023-03-30 16:56:10,141 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:56:10,144 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=164, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/adc2c50c0ded4264bfe95dabc31956c8 2023-03-30 16:56:10,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:10,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195430143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:10,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195430144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,195 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:10,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:10,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:10,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:10,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:10,199 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:10,199 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:10,199 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:10,199 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:10,200 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,200 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,201 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,201 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:10,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:10,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:10,201 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,201 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:10,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:10,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:10,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:10,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:10,203 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:10,203 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:10,203 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:10,203 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:10,205 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,205 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:10,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:10,206 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:10,206 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:10,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:10,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:10,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:10,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@785f7fdc[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:10,207 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:10,207 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,208 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,208 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:10,209 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:10,209 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:10,209 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,209 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:10,211 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,211 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:10,211 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:10,211 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:10,213 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:10,214 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:10,214 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:10,214 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:10,214 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:10,215 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,215 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:10,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,215 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:10,215 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:10,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:10,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:10,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:10,217 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,219 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:10,219 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:10,219 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,219 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:10,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:10,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:10,219 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,219 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,219 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,219 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,220 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,219 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:10,220 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:10,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:10,220 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:10,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:10,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:10,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:10,237 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-30 16:56:10,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:10,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195430249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:10,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195430251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,320 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:10,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:10,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:10,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:10,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:10,326 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:10,327 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:10,327 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:10,327 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:10,327 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,327 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,328 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,328 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:10,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:10,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:10,329 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,329 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:10,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:10,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:10,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:10,344 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:10,344 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:10,344 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:10,352 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:10,352 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:10,352 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:10,352 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:10,352 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:10,352 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:10,352 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:10,352 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:10,352 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,352 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:10,352 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:10,355 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:10,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:10,355 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:10,356 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,356 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:10,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:10,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:10,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:10,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@326bd5ab[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:10,358 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:10,358 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,359 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,359 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:10,359 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:10,359 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:10,360 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,360 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:10,361 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:10,361 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:10,361 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:10,361 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:10,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:10,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:10,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:10,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:10,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:10,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:10,365 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:10,365 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:10,365 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:10,365 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:10,365 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:10,367 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:10,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:10,367 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,367 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:10,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:10,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,368 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:10,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:10,370 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:10,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,373 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:10,373 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:10,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:10,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:10,374 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,374 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,374 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,374 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,374 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,374 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,374 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:10,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:10,374 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:10,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:10,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:10,376 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:10,376 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:10,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:10,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:10,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195430453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:10,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195430455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,476 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:10,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:10,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:10,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:10,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:10,480 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:10,480 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:10,480 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:10,480 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:10,480 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,480 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,482 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:10,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:10,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:10,482 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,483 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,483 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:10,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:10,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:10,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:10,484 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:10,484 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:10,484 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:10,488 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:10,488 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:10,488 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:10,488 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:10,488 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:10,488 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:10,488 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:10,488 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:10,488 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,488 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:10,488 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:10,490 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,490 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:10,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:10,491 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,491 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:10,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:10,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:10,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:10,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:10,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@739aec52[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:10,492 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:10,493 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,498 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,498 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:10,498 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:10,498 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:10,498 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,498 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:10,505 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:10,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:10,506 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:10,506 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:10,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:10,506 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:10,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:10,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:10,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:10,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:10,509 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:10,509 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:10,509 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:10,509 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:10,509 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:10,511 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,511 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,511 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:10,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:10,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:10,512 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:10,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:10,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:10,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:10,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,515 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:10,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:10,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,518 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,518 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,518 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,518 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,518 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:10,518 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:10,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:10,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:10,518 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,519 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:10,519 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:10,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:10,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:10,518 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:10,519 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:10,519 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:10,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:10,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:10,554 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/8237e19065f04e5aa2954199d6743ac8 2023-03-30 16:56:10,570 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/68ef3477e71a42dfb00f5740ab51bdbf 2023-03-30 16:56:10,574 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/adc2c50c0ded4264bfe95dabc31956c8 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/adc2c50c0ded4264bfe95dabc31956c8 2023-03-30 16:56:10,577 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/adc2c50c0ded4264bfe95dabc31956c8, entries=250, sequenceid=164, filesize=38.9 K 2023-03-30 16:56:10,578 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/8237e19065f04e5aa2954199d6743ac8 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/8237e19065f04e5aa2954199d6743ac8 2023-03-30 16:56:10,583 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/8237e19065f04e5aa2954199d6743ac8, entries=150, sequenceid=164, filesize=11.8 K 2023-03-30 16:56:10,584 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/68ef3477e71a42dfb00f5740ab51bdbf as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/68ef3477e71a42dfb00f5740ab51bdbf 2023-03-30 16:56:10,588 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/68ef3477e71a42dfb00f5740ab51bdbf, entries=150, sequenceid=164, filesize=11.8 K 2023-03-30 16:56:10,589 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 61fee4b7e4fa8ab90f65e22421bf6487 in 485ms, sequenceid=164, compaction requested=false 2023-03-30 16:56:10,589 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:10,619 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:10,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:10,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:10,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:10,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:10,623 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:10,623 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:10,623 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:10,623 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:10,623 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,623 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,625 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:10,625 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:10,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,626 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,626 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:10,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:10,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:10,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:10,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:10,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:10,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:10,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:10,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:10,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:10,628 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:10,628 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:10,629 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-30 16:56:10,629 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:10,629 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:10,629 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:10,629 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:10,629 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:10,629 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:10,670 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330a675a3f9e0e84de9b01fbbd98a47de5e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330a675a3f9e0e84de9b01fbbd98a47de5e 2023-03-30 16:56:10,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:10,758 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:10,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:10,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195430780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:10,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195430781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:10,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51170 deadline: 1680195430790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,794 DEBUG [Thread-3537] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4134 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:56:10,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:10,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51108 deadline: 1680195430807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:10,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51126 deadline: 1680195430807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,812 DEBUG [Thread-3539] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:56:10,813 DEBUG [Thread-3533] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:56:10,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:10,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195430885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:10,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:10,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195430886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,099 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=182, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/6dd2535bad444dcdbca14654403cf167 2023-03-30 16:56:11,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:11,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195431099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:11,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195431100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,113 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/c182b77a21c645faa3cd60e68f14e7a4 2023-03-30 16:56:11,124 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/dd97f1c25a5e43058b6a7e7d9c86422a 2023-03-30 16:56:11,129 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/6dd2535bad444dcdbca14654403cf167 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/6dd2535bad444dcdbca14654403cf167 2023-03-30 16:56:11,133 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/6dd2535bad444dcdbca14654403cf167, entries=150, sequenceid=182, filesize=25.3 K 2023-03-30 16:56:11,133 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/c182b77a21c645faa3cd60e68f14e7a4 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/c182b77a21c645faa3cd60e68f14e7a4 2023-03-30 16:56:11,137 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/c182b77a21c645faa3cd60e68f14e7a4, entries=150, sequenceid=182, filesize=11.8 K 2023-03-30 16:56:11,139 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/dd97f1c25a5e43058b6a7e7d9c86422a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/dd97f1c25a5e43058b6a7e7d9c86422a 2023-03-30 16:56:11,145 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/dd97f1c25a5e43058b6a7e7d9c86422a, entries=150, sequenceid=182, filesize=11.8 K 2023-03-30 16:56:11,146 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 61fee4b7e4fa8ab90f65e22421bf6487 in 518ms, sequenceid=182, compaction requested=true 2023-03-30 16:56:11,147 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:11,147 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:11,147 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:11,147 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:11,147 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,147 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:11,147 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:11,149 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,149 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:11,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:11,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:11,149 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,149 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:11,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:11,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:11,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:11,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fbcdfd1[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:11,151 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:11,151 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,153 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,153 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:11,153 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:11,153 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:11,153 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,153 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:11,154 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:11,155 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:11,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:11,155 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:11,155 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:11,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:11,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:11,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:11,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:11,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:11,159 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:11,159 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:11,159 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:11,159 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:11,159 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:11,160 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,160 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:11,161 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:11,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:11,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:11,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:11,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:11,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,162 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:11,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,163 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:11,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,167 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,167 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,168 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,168 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,168 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:11,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:11,168 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:11,168 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:11,168 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:11,168 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,168 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,168 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:11,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:11,168 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:11,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:11,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:11,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:11,268 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:11,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:11,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:11,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:11,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:11,272 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:11,272 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:11,272 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:11,272 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:11,273 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,273 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,281 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:11,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:11,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:11,281 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,282 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,282 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:11,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:11,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:11,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:11,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:11,283 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:11,283 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:11,283 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:11,292 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:11,292 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:11,296 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:11,296 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:11,296 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-30 16:56:11,297 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:11,297 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:11,297 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:11,297 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:11,297 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:11,298 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:11,340 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/66c90f31481447da9907c4940fa17e10 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/66c90f31481447da9907c4940fa17e10 2023-03-30 16:56:11,341 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303305670d1f104834d688b515b9933745b91 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303305670d1f104834d688b515b9933745b91 2023-03-30 16:56:11,352 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61fee4b7e4fa8ab90f65e22421bf6487/A of 61fee4b7e4fa8ab90f65e22421bf6487 into 66c90f31481447da9907c4940fa17e10(size=25.6 K), total size for store is 89.8 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-30 16:56:11,352 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:11,352 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., storeName=61fee4b7e4fa8ab90f65e22421bf6487/A, priority=12, startTime=1680195370065; duration=1sec 2023-03-30 16:56:11,353 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:56:11,355 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=197, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/0b9ea9f393724802ac6d1078e5b7b733 2023-03-30 16:56:11,374 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/c753488bb5ac410b88f6235ef4cc95a3 2023-03-30 16:56:11,390 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/c1dd091701cb4bff9cb6538d9727b3b0 2023-03-30 16:56:11,397 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/0b9ea9f393724802ac6d1078e5b7b733 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/0b9ea9f393724802ac6d1078e5b7b733 2023-03-30 16:56:11,400 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/0b9ea9f393724802ac6d1078e5b7b733, entries=150, sequenceid=197, filesize=25.3 K 2023-03-30 16:56:11,400 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/c753488bb5ac410b88f6235ef4cc95a3 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/c753488bb5ac410b88f6235ef4cc95a3 2023-03-30 16:56:11,405 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/c753488bb5ac410b88f6235ef4cc95a3, entries=150, sequenceid=197, filesize=11.8 K 2023-03-30 16:56:11,405 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/c1dd091701cb4bff9cb6538d9727b3b0 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/c1dd091701cb4bff9cb6538d9727b3b0 2023-03-30 16:56:11,414 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/c1dd091701cb4bff9cb6538d9727b3b0, entries=150, sequenceid=197, filesize=11.8 K 2023-03-30 16:56:11,415 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 61fee4b7e4fa8ab90f65e22421bf6487 in 119ms, sequenceid=197, compaction requested=true 2023-03-30 16:56:11,415 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:11,415 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:11,415 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:11,415 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:11,415 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,415 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:11,415 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:11,417 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:11,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:11,417 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:11,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:11,418 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,418 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:11,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:11,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:11,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c1ef465[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:11,419 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:11,419 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,420 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,420 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:11,420 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:11,420 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:11,420 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,421 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:11,422 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:11,422 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:11,422 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,422 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:11,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:11,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:11,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:11,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:11,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:11,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:11,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:11,424 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:11,425 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:11,425 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:11,425 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:11,425 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:11,427 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,427 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:11,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:11,427 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:11,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:11,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:11,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:11,428 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:11,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,429 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:11,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,431 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:11,431 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:11,431 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,431 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,431 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:11,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:11,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:11,431 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:11,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:11,432 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,432 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,432 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,432 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:11,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:11,432 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:11,432 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:11,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:11,477 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:56:11,478 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:11,478 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:11,478 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:11,478 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:11,478 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:11,478 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:11,532 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:11,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:11,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:11,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:11,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:11,535 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:11,535 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:11,535 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:11,535 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:11,536 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,536 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,537 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:11,537 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:11,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:11,537 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,537 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:11,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:11,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:11,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:11,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:11,539 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:11,539 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:11,539 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:11,539 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:11,539 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:11,539 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:11,539 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:11,539 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:11,539 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:11,539 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:11,539 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:11,539 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,539 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:11,539 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:11,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195371528, max timestamp=1680195371538Num uniques 859; 2023-03-30 16:56:11,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:56:11,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680195371528, max timestamp=1680195371538Num uniques 859; 2023-03-30 16:56:11,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:56:11,542 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,542 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:11,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:11,542 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:11,543 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:11,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:11,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:11,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:11,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40b4aac0[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:11,544 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:11,544 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,546 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,547 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:11,547 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:11,547 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,547 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:11,547 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:11,548 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,548 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:11,548 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:11,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:11,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:11,548 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:11,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:11,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:11,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:11,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:11,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:11,551 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:11,551 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:11,551 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:11,551 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:11,551 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:11,560 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,560 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,560 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:11,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:11,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:11,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:11,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:11,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:11,561 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:11,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,562 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:11,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,564 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:11,564 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:11,564 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:11,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:11,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:11,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:11,564 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:11,564 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:11,565 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:11,564 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:11,565 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,565 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:11,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:11,565 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,565 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,565 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:11,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195431565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:11,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195431566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,665 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:11,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:11,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:11,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:11,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:11,668 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:11,668 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:11,668 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:11,668 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:11,668 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,668 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,669 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:11,670 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:11,670 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,670 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:11,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:11,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:11,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:11,671 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:11,671 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:11,671 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:11,672 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:11,672 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:11,672 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:11,672 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:11,672 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:11,672 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:11,672 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:11,672 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:11,672 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,672 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:11,672 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:11,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:11,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195431671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:11,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51118 deadline: 1680195431671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,675 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:11,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:11,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,675 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:11,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:11,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:11,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:11,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:11,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64e836f8[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:11,677 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:11,677 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,678 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,678 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:11,678 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:11,678 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:11,678 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,678 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:11,679 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,679 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:11,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:11,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:11,680 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:11,680 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:11,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:11,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:11,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:11,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:11,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:11,682 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:11,682 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:11,682 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:11,682 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:11,682 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:11,683 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,683 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:11,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,683 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:11,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:11,684 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:11,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:11,685 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:11,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,689 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,689 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,689 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,689 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,689 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:11,689 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:11,689 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:11,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:11,689 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,689 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:11,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:11,689 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:11,689 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:11,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:11,689 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:11,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:11,789 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:11,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:11,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:11,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:11,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:11,792 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:11,793 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:11,793 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:11,793 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:11,793 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,793 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,794 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:11,794 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:11,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:11,795 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,795 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:11,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:11,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:11,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:11,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:11,796 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:11,796 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:11,796 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:11,796 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:11,796 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:11,796 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:11,796 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:11,796 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:11,796 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:11,796 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:11,796 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:11,796 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,797 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:11,797 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:11,798 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,798 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:11,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:11,799 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:11,799 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:11,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:11,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:11,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:11,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2de32fe3[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:11,801 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:11,801 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,802 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,803 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:11,803 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:11,803 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:11,803 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,803 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:11,805 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:11,805 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:11,805 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:11,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:11,805 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:11,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:11,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:11,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:11,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:11,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:11,808 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:11,808 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:11,808 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:11,808 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:11,808 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:11,809 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,809 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:11,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:11,809 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:11,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:11,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:11,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:11,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,810 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:11,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,811 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,814 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,814 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,814 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,814 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,814 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,814 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,814 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:11,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:11,814 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:11,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:11,814 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:11,814 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:11,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:11,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:11,814 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:11,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:11,815 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:11,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:11,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195431875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:11,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51160 deadline: 1680195431875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,898 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303306dbe507cb33f4ef6b8a9ea6dede21401 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303306dbe507cb33f4ef6b8a9ea6dede21401 2023-03-30 16:56:11,912 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=209, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/e7d2fa26e49543498550b8823efa1cfb 2023-03-30 16:56:11,915 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:11,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:11,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:11,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:11,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:11,918 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:11,918 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:11,919 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:11,919 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:11,919 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,919 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,924 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:11,924 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:11,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:11,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,924 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,924 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:11,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:11,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:11,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:11,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:11,928 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:11,928 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:11,928 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:11,936 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:11,936 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:11,936 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/826730381a8943358b13062147194b1d 2023-03-30 16:56:11,936 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:11,936 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:11,936 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:11,936 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:11,936 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:11,936 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:11,936 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,937 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:11,937 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:11,940 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,940 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:11,941 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,941 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:11,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:11,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:11,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:11,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:11,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76054be6[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:11,944 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:11,944 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,945 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,946 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:11,946 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:11,946 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:11,946 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,946 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:11,948 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:11,948 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:11,948 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:11,948 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:11,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:11,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:11,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:11,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:11,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:11,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:11,951 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:11,951 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:11,951 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:11,951 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:11,951 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:11,953 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,953 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:11,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,953 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:11,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:11,954 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/f907f74f05c641658fba360bf6fc66db 2023-03-30 16:56:11,958 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/e7d2fa26e49543498550b8823efa1cfb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/e7d2fa26e49543498550b8823efa1cfb 2023-03-30 16:56:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:11,962 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/e7d2fa26e49543498550b8823efa1cfb, entries=200, sequenceid=209, filesize=32.1 K 2023-03-30 16:56:11,963 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/826730381a8943358b13062147194b1d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/826730381a8943358b13062147194b1d 2023-03-30 16:56:11,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,964 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,966 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:11,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,967 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/826730381a8943358b13062147194b1d, entries=150, sequenceid=209, filesize=11.8 K 2023-03-30 16:56:11,968 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/f907f74f05c641658fba360bf6fc66db as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/f907f74f05c641658fba360bf6fc66db 2023-03-30 16:56:11,971 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/f907f74f05c641658fba360bf6fc66db, entries=150, sequenceid=209, filesize=11.8 K 2023-03-30 16:56:11,972 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:11,972 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:11,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:11,972 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=123.00 KB/125950 for 61fee4b7e4fa8ab90f65e22421bf6487 in 495ms, sequenceid=209, compaction requested=true 2023-03-30 16:56:11,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:11,972 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:11,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:11,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:11,972 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:56:11,972 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 16:56:11,972 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:56:11,972 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:11,972 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,973 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:11,973 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,973 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:11,973 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:11,973 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,973 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:11,973 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:11,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:11,972 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:56:11,972 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 16:56:11,973 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:11,974 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 150728 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 16:56:11,974 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 61fee4b7e4fa8ab90f65e22421bf6487/A is initiating minor compaction (all files) 2023-03-30 16:56:11,974 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61fee4b7e4fa8ab90f65e22421bf6487/A in TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:11,974 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/66c90f31481447da9907c4940fa17e10, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/adc2c50c0ded4264bfe95dabc31956c8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/6dd2535bad444dcdbca14654403cf167, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/0b9ea9f393724802ac6d1078e5b7b733, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/e7d2fa26e49543498550b8823efa1cfb] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp, totalSize=147.2 K 2023-03-30 16:56:11,974 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60454 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 16:56:11,974 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 61fee4b7e4fa8ab90f65e22421bf6487/B is initiating minor compaction (all files) 2023-03-30 16:56:11,974 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61fee4b7e4fa8ab90f65e22421bf6487/B in TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:11,974 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/4cbafb90084a40b2a4d136b914653234, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/8237e19065f04e5aa2954199d6743ac8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/c182b77a21c645faa3cd60e68f14e7a4, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/c753488bb5ac410b88f6235ef4cc95a3, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/826730381a8943358b13062147194b1d] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp, totalSize=59.0 K 2023-03-30 16:56:11,975 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 66c90f31481447da9907c4940fa17e10, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1680195369868 2023-03-30 16:56:11,975 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 4cbafb90084a40b2a4d136b914653234, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1680195369868 2023-03-30 16:56:11,975 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting adc2c50c0ded4264bfe95dabc31956c8, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1680195369989 2023-03-30 16:56:11,975 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 8237e19065f04e5aa2954199d6743ac8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1680195369989 2023-03-30 16:56:11,976 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 6dd2535bad444dcdbca14654403cf167, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680195370132 2023-03-30 16:56:11,976 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting c182b77a21c645faa3cd60e68f14e7a4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680195370132 2023-03-30 16:56:11,976 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 0b9ea9f393724802ac6d1078e5b7b733, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680195370771 2023-03-30 16:56:11,976 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting c753488bb5ac410b88f6235ef4cc95a3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680195370771 2023-03-30 16:56:11,976 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting e7d2fa26e49543498550b8823efa1cfb, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680195371459 2023-03-30 16:56:11,976 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 826730381a8943358b13062147194b1d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680195371469 2023-03-30 16:56:11,986 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 61fee4b7e4fa8ab90f65e22421bf6487#B#compaction#497 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:56:11,993 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61fee4b7e4fa8ab90f65e22421bf6487#A#compaction#498 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:56:12,018 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/39fce1ac08b94888a6013d543dae46eb as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/39fce1ac08b94888a6013d543dae46eb 2023-03-30 16:56:12,022 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 61fee4b7e4fa8ab90f65e22421bf6487/A of 61fee4b7e4fa8ab90f65e22421bf6487 into 39fce1ac08b94888a6013d543dae46eb(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:56:12,022 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:12,022 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., storeName=61fee4b7e4fa8ab90f65e22421bf6487/A, priority=11, startTime=1680195371972; duration=0sec 2023-03-30 16:56:12,022 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:56:12,022 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-30 16:56:12,024 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60454 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-30 16:56:12,024 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 61fee4b7e4fa8ab90f65e22421bf6487/C is initiating minor compaction (all files) 2023-03-30 16:56:12,025 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61fee4b7e4fa8ab90f65e22421bf6487/C in TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:12,025 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/4f253933dbba40e9b915944427cadec3, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/68ef3477e71a42dfb00f5740ab51bdbf, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/dd97f1c25a5e43058b6a7e7d9c86422a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/c1dd091701cb4bff9cb6538d9727b3b0, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/f907f74f05c641658fba360bf6fc66db] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp, totalSize=59.0 K 2023-03-30 16:56:12,025 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 4f253933dbba40e9b915944427cadec3, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1680195369868 2023-03-30 16:56:12,025 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 68ef3477e71a42dfb00f5740ab51bdbf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1680195369989 2023-03-30 16:56:12,025 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting dd97f1c25a5e43058b6a7e7d9c86422a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680195370132 2023-03-30 16:56:12,026 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting c1dd091701cb4bff9cb6538d9727b3b0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1680195370771 2023-03-30 16:56:12,026 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting f907f74f05c641658fba360bf6fc66db, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680195371469 2023-03-30 16:56:12,034 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61fee4b7e4fa8ab90f65e22421bf6487#C#compaction#499 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:56:12,073 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:12,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:12,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:12,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:12,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:12,076 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:12,077 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:12,077 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:12,077 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:12,077 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,077 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,078 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:12,078 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:12,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:12,079 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,079 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:12,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:12,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:12,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:12,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:12,080 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:12,080 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:12,080 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:12,080 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:12,080 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:12,080 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:12,080 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:12,080 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=123.00 KB heapSize=353.63 KB 2023-03-30 16:56:12,081 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:12,081 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:12,081 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:12,081 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:12,081 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:12,081 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:12,096 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033021796a71e1204e39812886cf1eda723c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033021796a71e1204e39812886cf1eda723c 2023-03-30 16:56:12,103 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=240, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/2e84363205134912876b76efc895f657 2023-03-30 16:56:12,113 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/c0c25d0116d041f0955138c9c9638754 2023-03-30 16:56:12,123 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/dbf8d35cdc86400f998f8d5b2db87da6 2023-03-30 16:56:12,126 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/2e84363205134912876b76efc895f657 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/2e84363205134912876b76efc895f657 2023-03-30 16:56:12,130 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/2e84363205134912876b76efc895f657, entries=150, sequenceid=240, filesize=25.3 K 2023-03-30 16:56:12,130 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/c0c25d0116d041f0955138c9c9638754 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/c0c25d0116d041f0955138c9c9638754 2023-03-30 16:56:12,133 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/c0c25d0116d041f0955138c9c9638754, entries=150, sequenceid=240, filesize=11.8 K 2023-03-30 16:56:12,134 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/dbf8d35cdc86400f998f8d5b2db87da6 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/dbf8d35cdc86400f998f8d5b2db87da6 2023-03-30 16:56:12,137 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/dbf8d35cdc86400f998f8d5b2db87da6, entries=150, sequenceid=240, filesize=11.8 K 2023-03-30 16:56:12,138 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~123.00 KB/125950, heapSize ~353.58 KB/362064, currentSize=0 B/0 for 61fee4b7e4fa8ab90f65e22421bf6487 in 58ms, sequenceid=240, compaction requested=false 2023-03-30 16:56:12,138 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:12,138 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:12,138 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:12,138 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:12,138 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,138 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:12,138 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:12,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,140 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:12,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:12,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13820d4c[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:12,144 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:12,144 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,146 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:12,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:12,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:12,146 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,146 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:12,147 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:12,147 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,147 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:12,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,147 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:12,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:12,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:12,151 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:12,151 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:12,151 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:12,151 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:12,151 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:12,152 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,152 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:12,152 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:12,153 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,154 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,156 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,156 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,156 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,156 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,156 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:12,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:12,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:12,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:12,157 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:12,157 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:12,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:12,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:12,157 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:12,157 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:12,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:12,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:12,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:12,240 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:56:12,240 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:12,240 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:12,240 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:12,240 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:12,240 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:12,241 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:12,257 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:12,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:12,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:12,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:12,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:12,260 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:12,260 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:12,260 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:12,260 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:12,261 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,261 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,264 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:12,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:12,264 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:12,264 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,264 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:12,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:12,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:12,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:12,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:12,266 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:12,266 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:12,266 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:12,272 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:12,272 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:12,272 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:12,272 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:12,272 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:12,272 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:12,272 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:12,272 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:12,272 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,272 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:12,272 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:12,274 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,274 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,275 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,275 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:12,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36f8a632[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:12,276 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:12,276 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,278 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,278 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:12,278 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:12,278 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:12,278 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,278 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:12,279 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,279 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:12,279 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:12,279 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:12,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:12,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:12,282 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:12,282 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:12,282 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:12,282 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:12,282 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:12,289 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,289 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,289 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:12,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:12,290 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195372279, max timestamp=1680195372288Num uniques 859; 2023-03-30 16:56:12,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:56:12,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195372279, max timestamp=1680195372288Num uniques 859; 2023-03-30 16:56:12,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195372279, max timestamp=1680195372288Num uniques 859; 2023-03-30 16:56:12,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:56:12,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:56:12,291 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,294 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:12,294 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:12,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:12,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:12,295 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,295 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,295 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,295 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,295 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,295 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,295 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:12,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:12,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:12,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:12,295 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:12,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:12,295 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:12,295 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:12,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:12,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195432327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:12,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195432328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,395 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:12,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:12,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:12,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:12,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:12,399 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:12,400 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:12,400 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:12,400 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:12,400 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/e4805990270d4f3f98bfb0c43c924108 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e4805990270d4f3f98bfb0c43c924108 2023-03-30 16:56:12,400 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,400 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,402 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:12,402 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:12,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:12,402 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,402 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:12,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:12,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:12,403 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:12,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:12,403 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:12,403 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:12,404 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:12,404 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:12,404 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:12,404 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:12,404 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:12,405 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 61fee4b7e4fa8ab90f65e22421bf6487/B of 61fee4b7e4fa8ab90f65e22421bf6487 into e4805990270d4f3f98bfb0c43c924108(size=12.2 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:56:12,405 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:12,405 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., storeName=61fee4b7e4fa8ab90f65e22421bf6487/B, priority=11, startTime=1680195371972; duration=0sec 2023-03-30 16:56:12,405 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:56:12,407 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,407 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,408 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,408 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:12,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2500b39[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:12,409 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:12,409 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,411 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,411 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:12,412 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,412 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:12,412 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:12,412 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:12,413 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,414 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:12,414 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:12,414 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:12,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:12,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:12,416 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:12,416 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:12,416 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:12,416 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:12,416 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:12,418 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,418 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:12,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,419 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:12,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:12,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,419 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,422 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,428 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:12,428 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:12,428 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:12,428 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:12,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:12,428 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,428 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:12,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:12,428 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,428 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,428 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,428 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:12,428 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:12,428 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:12,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:12,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:12,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195432432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:12,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195432432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,448 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/64ead87b2e4d4778878b3f81e0c86adf as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/64ead87b2e4d4778878b3f81e0c86adf 2023-03-30 16:56:12,453 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 61fee4b7e4fa8ab90f65e22421bf6487/C of 61fee4b7e4fa8ab90f65e22421bf6487 into 64ead87b2e4d4778878b3f81e0c86adf(size=12.2 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:56:12,453 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:12,453 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., storeName=61fee4b7e4fa8ab90f65e22421bf6487/C, priority=11, startTime=1680195371972; duration=0sec 2023-03-30 16:56:12,453 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:56:12,529 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:12,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:12,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:12,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:12,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:12,532 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:12,532 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:12,532 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:12,532 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:12,532 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,532 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,534 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:12,534 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:12,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:12,534 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,534 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:12,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:12,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:12,535 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:12,535 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:12,535 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:12,535 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:12,535 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:12,535 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:12,535 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:12,535 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:12,536 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:12,536 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:12,536 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:12,536 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:12,536 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:12,536 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,536 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:12,536 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:12,537 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,537 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,538 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,538 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:12,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:12,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e562aee[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:12,539 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:12,539 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,541 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,541 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:12,542 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:12,542 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:12,542 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,542 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:12,543 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:12,543 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:12,543 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,543 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:12,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:12,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:12,546 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:12,546 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:12,546 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:12,546 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:12,546 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:12,547 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,547 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,547 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:12,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:12,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:12,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,549 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,550 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,552 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:12,552 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:12,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:12,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:12,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:12,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:12,552 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:12,552 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,552 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,552 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,552 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,552 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,552 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,552 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:12,553 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:12,553 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:12,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:12,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:12,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:12,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195432636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:12,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195432636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,653 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:12,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:12,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:12,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:12,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:12,659 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:12,659 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:12,659 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:12,659 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:12,660 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,660 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,661 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033039bbac195a8745d7b44c081c2ba83ea1 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033039bbac195a8745d7b44c081c2ba83ea1 2023-03-30 16:56:12,665 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,665 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:12,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:12,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:12,666 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,666 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:12,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:12,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:12,668 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:12,668 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:12,668 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:12,668 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=251, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/6d960c15ed22434a9c58cc1ed30ebb81 2023-03-30 16:56:12,668 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:12,668 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:12,668 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:12,668 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:12,668 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:12,669 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:12,669 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:12,669 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:12,669 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,669 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:12,669 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:12,671 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,671 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,671 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,671 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:12,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dbda405[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:12,674 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:12,674 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,675 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,676 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:12,676 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:12,676 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:12,676 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,676 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:12,677 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,677 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:12,678 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:12,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,678 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:12,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:12,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:12,680 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:12,680 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:12,680 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:12,680 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:12,680 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:12,682 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,682 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:12,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,682 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,683 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,685 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,688 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,688 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,688 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,688 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,688 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,688 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,688 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,688 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:12,688 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:12,688 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:12,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:12,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:12,689 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:12,689 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:12,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:12,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:12,789 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:12,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:12,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:12,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:12,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:12,793 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:12,793 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:12,793 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:12,793 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:12,793 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,793 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,795 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,795 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:12,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:12,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:12,795 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,795 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:12,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:12,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:12,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:12,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:12,797 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:12,797 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:12,797 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:12,797 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:12,797 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:12,797 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:12,797 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:12,797 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:12,797 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:12,797 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:12,797 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:12,797 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,797 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:12,797 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:12,799 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,799 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,799 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,800 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:12,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e8750b6[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:12,801 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:12,801 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,802 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,803 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:12,803 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:12,803 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:12,803 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,803 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:12,804 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:12,804 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:12,804 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,804 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:12,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:12,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:12,809 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:12,809 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:12,809 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:12,809 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:12,809 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:12,811 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,811 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:12,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,811 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,812 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,813 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,814 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,814 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,814 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,814 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,814 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,814 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,814 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,814 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:12,814 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:12,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:12,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:12,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:12,815 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:12,815 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:12,815 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:12,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:12,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:12,915 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:12,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:12,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:12,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:12,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:12,918 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:12,919 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:12,919 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:12,919 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:12,919 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,919 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,921 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,921 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:12,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:12,922 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,922 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:12,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:12,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:12,923 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:12,923 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:12,923 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:12,923 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:12,923 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:12,923 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:12,923 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:12,923 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:12,924 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:12,924 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:12,924 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:12,924 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,924 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:12,924 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:12,928 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,928 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,929 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,929 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:12,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c0c8f9c[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:12,931 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:12,931 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,932 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,932 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:12,932 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:12,932 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:12,932 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,932 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:12,933 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,933 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:12,934 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:12,934 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:12,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:12,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:12,936 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:12,936 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:12,936 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:12,936 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:12,936 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:12,939 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,939 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,939 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:12,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:12,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,940 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:12,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195432939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,941 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:12,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195432941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,944 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,944 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,944 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:12,944 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:12,944 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:12,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:12,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:12,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:12,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:12,944 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:12,944 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:12,945 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:12,945 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:12,945 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,945 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:12,945 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:12,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:12,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:12,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:13,045 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:13,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:13,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:13,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:13,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:13,048 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:13,048 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:13,048 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:13,048 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:13,049 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,049 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,050 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:13,050 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:13,051 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,051 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:13,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:13,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:13,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:13,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:13,052 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:13,052 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:13,052 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:13,052 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:13,052 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:13,052 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:13,052 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:13,052 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:13,052 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:13,052 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:13,053 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:13,053 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,053 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:13,053 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:13,054 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,055 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:13,055 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,055 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:13,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:13,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43e2d1a9[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:13,057 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:13,057 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,059 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,060 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:13,060 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:13,060 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:13,060 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,060 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:13,061 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,061 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:13,062 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:13,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:13,062 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:13,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:13,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:13,064 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:13,064 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:13,064 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:13,064 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:13,065 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:13,066 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,066 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,066 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:13,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:13,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:13,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:13,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:13,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:13,067 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,069 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,071 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,071 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,071 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,071 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,071 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,071 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:13,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:13,071 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:13,071 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:13,071 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:13,071 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:13,071 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,071 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:13,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:13,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:13,098 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/13e5399f81524f64ba0767404859a7aa 2023-03-30 16:56:13,172 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:13,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:13,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:13,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:13,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:13,175 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:13,175 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:13,175 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:13,175 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:13,176 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,176 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,177 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:13,178 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:13,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:13,178 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,178 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:13,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:13,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:13,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:13,179 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:13,179 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:13,179 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:13,179 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:13,180 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:13,179 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:13,180 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:13,180 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:13,180 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:13,180 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:13,180 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:13,180 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,180 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:13,180 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:13,181 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,181 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:13,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:13,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,182 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:13,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:13,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:13,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:13,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:13,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73cfd85c[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:13,184 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:13,184 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,186 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,186 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:13,186 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:13,186 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:13,186 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,186 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:13,187 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,187 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:13,187 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:13,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:13,187 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:13,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:13,190 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:13,190 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:13,190 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:13,190 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:13,190 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:13,193 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,193 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:13,193 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:13,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:13,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,193 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:13,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:13,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:13,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:13,195 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,197 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,197 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,197 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,197 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,197 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:13,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:13,197 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:13,197 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:13,197 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,197 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:13,197 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:13,197 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:13,197 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:13,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:13,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:13,297 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:13,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:13,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:13,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:13,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:13,301 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:13,301 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:13,301 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:13,301 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:13,301 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,301 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,303 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:13,303 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:13,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:13,303 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,303 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:13,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:13,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:13,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:13,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:13,304 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:13,304 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:13,304 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:13,304 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:13,304 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:13,305 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:13,304 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:13,305 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:13,305 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:13,305 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:13,305 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:13,305 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,305 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:13,305 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:13,306 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,306 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:13,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:13,307 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,307 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:13,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:13,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:13,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:13,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67593473[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:13,309 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:13,309 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,314 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,314 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:13,314 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:13,314 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,314 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:13,314 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:13,316 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,316 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:13,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:13,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:13,316 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:13,316 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:13,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:13,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:13,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:13,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:13,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:13,318 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:13,318 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:13,318 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:13,318 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:13,318 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:13,320 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,320 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,320 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:13,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:13,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:13,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:13,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:13,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:13,321 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:13,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,322 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:13,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,325 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:13,325 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:13,325 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:13,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:13,325 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,326 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:13,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:13,326 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:13,326 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,326 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:13,326 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,326 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:13,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:13,326 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,326 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,426 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:13,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:13,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:13,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:13,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:13,429 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:13,429 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:13,429 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:13,429 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:13,430 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,430 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,431 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:13,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:13,431 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,432 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,432 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:13,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:13,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:13,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:13,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:13,432 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:13,433 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:13,433 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:13,433 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:13,433 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:13,433 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:13,433 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:13,433 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:13,433 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:13,433 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:13,433 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:13,433 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,433 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:13,433 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:13,434 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:13,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:13,435 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:13,435 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,435 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:13,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:13,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:13,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:13,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f61376b[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:13,437 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:13,437 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,438 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,439 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:13,439 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:13,439 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:13,439 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,439 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:13,441 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,441 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:13,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:13,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:13,441 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:13,441 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:13,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:13,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:13,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:13,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:13,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:13,443 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:13,444 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:13,444 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:13,444 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:13,444 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:13,445 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,445 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:13,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:13,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,446 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:13,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:13,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195433444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:13,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:13,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:13,447 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:13,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,448 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,451 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:13,451 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,451 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:13,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:13,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:13,451 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:13,451 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:13,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:13,451 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,451 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:13,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:13,451 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:13,451 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,451 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:13,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195433447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,452 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:13,452 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,512 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/9e6305984d0c4ffaa873b7900f950280 2023-03-30 16:56:13,516 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/6d960c15ed22434a9c58cc1ed30ebb81 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/6d960c15ed22434a9c58cc1ed30ebb81 2023-03-30 16:56:13,519 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/6d960c15ed22434a9c58cc1ed30ebb81, entries=300, sequenceid=251, filesize=45.8 K 2023-03-30 16:56:13,520 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/13e5399f81524f64ba0767404859a7aa as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/13e5399f81524f64ba0767404859a7aa 2023-03-30 16:56:13,523 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/13e5399f81524f64ba0767404859a7aa, entries=150, sequenceid=251, filesize=11.8 K 2023-03-30 16:56:13,524 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/9e6305984d0c4ffaa873b7900f950280 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/9e6305984d0c4ffaa873b7900f950280 2023-03-30 16:56:13,527 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/9e6305984d0c4ffaa873b7900f950280, entries=150, sequenceid=251, filesize=11.8 K 2023-03-30 16:56:13,527 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 61fee4b7e4fa8ab90f65e22421bf6487 in 1287ms, sequenceid=251, compaction requested=true 2023-03-30 16:56:13,528 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:13,528 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:56:13,528 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:56:13,528 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:56:13,528 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:56:13,528 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:56:13,529 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 99106 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:56:13,529 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 61fee4b7e4fa8ab90f65e22421bf6487/A is initiating minor compaction (all files) 2023-03-30 16:56:13,529 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61fee4b7e4fa8ab90f65e22421bf6487/A in TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:13,529 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/39fce1ac08b94888a6013d543dae46eb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/2e84363205134912876b76efc895f657, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/6d960c15ed22434a9c58cc1ed30ebb81] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp, totalSize=96.8 K 2023-03-30 16:56:13,529 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:56:13,529 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 39fce1ac08b94888a6013d543dae46eb, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680195371469 2023-03-30 16:56:13,529 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 61fee4b7e4fa8ab90f65e22421bf6487/B is initiating minor compaction (all files) 2023-03-30 16:56:13,530 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61fee4b7e4fa8ab90f65e22421bf6487/B in TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:13,530 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e4805990270d4f3f98bfb0c43c924108, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/c0c25d0116d041f0955138c9c9638754, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/13e5399f81524f64ba0767404859a7aa] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp, totalSize=35.7 K 2023-03-30 16:56:13,530 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 2e84363205134912876b76efc895f657, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680195371550 2023-03-30 16:56:13,530 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting e4805990270d4f3f98bfb0c43c924108, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680195371469 2023-03-30 16:56:13,530 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 6d960c15ed22434a9c58cc1ed30ebb81, keycount=300, bloomtype=ROW, size=45.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680195372223 2023-03-30 16:56:13,530 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting c0c25d0116d041f0955138c9c9638754, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680195371550 2023-03-30 16:56:13,531 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 13e5399f81524f64ba0767404859a7aa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680195372233 2023-03-30 16:56:13,536 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 61fee4b7e4fa8ab90f65e22421bf6487#B#compaction#507 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:56:13,541 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61fee4b7e4fa8ab90f65e22421bf6487#A#compaction#506 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:56:13,552 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:13,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:13,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:13,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:13,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:13,556 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:13,556 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:13,556 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:13,556 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:13,556 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,557 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,557 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/820b7d3d7a92476cbb35b38f630146ac as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/820b7d3d7a92476cbb35b38f630146ac 2023-03-30 16:56:13,558 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,558 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:13,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:13,558 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:13,558 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:13,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:13,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:13,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:13,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:13,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:13,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:13,560 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:13,560 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:13,560 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:13,562 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61fee4b7e4fa8ab90f65e22421bf6487/B of 61fee4b7e4fa8ab90f65e22421bf6487 into 820b7d3d7a92476cbb35b38f630146ac(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:56:13,562 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:13,562 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., storeName=61fee4b7e4fa8ab90f65e22421bf6487/B, priority=13, startTime=1680195373528; duration=0sec 2023-03-30 16:56:13,562 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:56:13,562 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-30 16:56:13,563 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-30 16:56:13,563 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 61fee4b7e4fa8ab90f65e22421bf6487/C is initiating minor compaction (all files) 2023-03-30 16:56:13,563 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61fee4b7e4fa8ab90f65e22421bf6487/C in TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:13,563 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/64ead87b2e4d4778878b3f81e0c86adf, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/dbf8d35cdc86400f998f8d5b2db87da6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/9e6305984d0c4ffaa873b7900f950280] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp, totalSize=35.7 K 2023-03-30 16:56:13,564 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 64ead87b2e4d4778878b3f81e0c86adf, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1680195371469 2023-03-30 16:56:13,564 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:13,564 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:13,564 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:13,564 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:13,564 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting dbf8d35cdc86400f998f8d5b2db87da6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680195371550 2023-03-30 16:56:13,564 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-30 16:56:13,564 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:13,564 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:13,564 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:13,564 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 9e6305984d0c4ffaa873b7900f950280, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680195372233 2023-03-30 16:56:13,564 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:13,565 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:13,565 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:13,569 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 61fee4b7e4fa8ab90f65e22421bf6487#C#compaction#508 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:56:13,971 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/52d7384513784d59af083d9480a80108 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/52d7384513784d59af083d9480a80108 2023-03-30 16:56:13,989 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61fee4b7e4fa8ab90f65e22421bf6487/A of 61fee4b7e4fa8ab90f65e22421bf6487 into 52d7384513784d59af083d9480a80108(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:56:13,989 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:13,989 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., storeName=61fee4b7e4fa8ab90f65e22421bf6487/A, priority=13, startTime=1680195373528; duration=0sec 2023-03-30 16:56:13,990 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:56:13,994 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/9a72d3cb776340e7ac05a4a88456ebba as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/9a72d3cb776340e7ac05a4a88456ebba 2023-03-30 16:56:13,999 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61fee4b7e4fa8ab90f65e22421bf6487/C of 61fee4b7e4fa8ab90f65e22421bf6487 into 9a72d3cb776340e7ac05a4a88456ebba(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:56:13,999 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:13,999 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., storeName=61fee4b7e4fa8ab90f65e22421bf6487/C, priority=13, startTime=1680195373528; duration=0sec 2023-03-30 16:56:13,999 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:56:14,008 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303305c46970c27ee421f9de329edb7411071 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303305c46970c27ee421f9de329edb7411071 2023-03-30 16:56:14,422 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=290, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/7be553d9370d4a3a9fe5f687c483588c 2023-03-30 16:56:14,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:14,454 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:14,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:14,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195434466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:14,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195434469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:14,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195434569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:14,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195434572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:14,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195434774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:14,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195434776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:14,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51170 deadline: 1680195434818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,821 DEBUG [Thread-3537] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:56:14,836 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/cfa2c71eb1c8491382fb339066b01f78 2023-03-30 16:56:14,846 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/d7605159ca9a40229b1476c4260caa8a 2023-03-30 16:56:14,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:14,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51108 deadline: 1680195434848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:14,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51126 deadline: 1680195434848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,852 DEBUG [Thread-3533] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8192 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:56:14,853 DEBUG [Thread-3539] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8190 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:56:14,853 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/7be553d9370d4a3a9fe5f687c483588c as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/7be553d9370d4a3a9fe5f687c483588c 2023-03-30 16:56:14,857 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/7be553d9370d4a3a9fe5f687c483588c, entries=150, sequenceid=290, filesize=25.4 K 2023-03-30 16:56:14,857 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/cfa2c71eb1c8491382fb339066b01f78 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/cfa2c71eb1c8491382fb339066b01f78 2023-03-30 16:56:14,861 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/cfa2c71eb1c8491382fb339066b01f78, entries=150, sequenceid=290, filesize=11.9 K 2023-03-30 16:56:14,861 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/d7605159ca9a40229b1476c4260caa8a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/d7605159ca9a40229b1476c4260caa8a 2023-03-30 16:56:14,865 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/d7605159ca9a40229b1476c4260caa8a, entries=150, sequenceid=290, filesize=11.9 K 2023-03-30 16:56:14,865 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 61fee4b7e4fa8ab90f65e22421bf6487 in 1301ms, sequenceid=290, compaction requested=false 2023-03-30 16:56:14,865 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:14,865 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:14,865 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:14,865 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:14,865 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,867 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:14,867 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:14,870 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,870 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:14,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:14,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:14,870 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:14,870 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:14,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:14,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:14,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:14,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:14,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:14,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7adbd4b7[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:14,872 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:14,872 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:14,874 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:14,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:14,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:14,874 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:14,874 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:14,874 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:14,874 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,874 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:14,875 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,875 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:14,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:14,875 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:14,876 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:14,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:14,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:14,878 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:14,878 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:14,878 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:14,878 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:14,879 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:14,880 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:14,880 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:14,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:14,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:14,880 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:14,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:14,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:14,881 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:14,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:14,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:14,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:14,882 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:14,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:14,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,885 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,885 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:14,886 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:14,886 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,886 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:14,885 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:14,886 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:14,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:14,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:14,886 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:14,886 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:14,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:14,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:14,886 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:14,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:14,886 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:14,886 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:14,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:14,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:14,986 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:14,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:14,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:14,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:14,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:14,993 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:14,993 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:14,993 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:14,993 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:14,994 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:14,994 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:14,995 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:14,995 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:14,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:14,995 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:14,995 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:14,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:14,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:14,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:14,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:14,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:14,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:14,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:14,996 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:14,997 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:14,997 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:14,997 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:14,997 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:14,997 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:14,997 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:14,997 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 16:56:14,997 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:14,997 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:14,997 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:14,997 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:14,997 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:14,997 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:15,022 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033058bbcbbab420440180a7d7028d644966 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033058bbcbbab420440180a7d7028d644966 2023-03-30 16:56:15,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:15,080 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:15,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195375100, max timestamp=1680195375104Num uniques 859; 2023-03-30 16:56:15,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:56:15,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195375100, max timestamp=1680195375104Num uniques 859; 2023-03-30 16:56:15,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195375100, max timestamp=1680195375104Num uniques 859; 2023-03-30 16:56:15,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:56:15,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:56:15,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:15,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195435132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:15,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195435133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:15,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195435236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:15,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195435237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,436 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=303, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/a9e4b5edfe7a4d3482e42d201752822e 2023-03-30 16:56:15,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:15,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195435438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:15,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195435441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,447 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/115839d6bdf64374bc56d36f1902edf4 2023-03-30 16:56:15,456 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/30ea69b733f84379b298af2bbc201a49 2023-03-30 16:56:15,460 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/a9e4b5edfe7a4d3482e42d201752822e as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/a9e4b5edfe7a4d3482e42d201752822e 2023-03-30 16:56:15,463 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/a9e4b5edfe7a4d3482e42d201752822e, entries=150, sequenceid=303, filesize=25.4 K 2023-03-30 16:56:15,464 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/115839d6bdf64374bc56d36f1902edf4 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/115839d6bdf64374bc56d36f1902edf4 2023-03-30 16:56:15,467 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/115839d6bdf64374bc56d36f1902edf4, entries=150, sequenceid=303, filesize=11.9 K 2023-03-30 16:56:15,468 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/30ea69b733f84379b298af2bbc201a49 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/30ea69b733f84379b298af2bbc201a49 2023-03-30 16:56:15,471 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/30ea69b733f84379b298af2bbc201a49, entries=150, sequenceid=303, filesize=11.9 K 2023-03-30 16:56:15,471 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 61fee4b7e4fa8ab90f65e22421bf6487 in 474ms, sequenceid=303, compaction requested=true 2023-03-30 16:56:15,472 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:15,472 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:15,472 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:15,472 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:15,472 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,472 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:15,472 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:15,473 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,473 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:15,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:15,474 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:15,474 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:15,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:15,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:15,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:15,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:15,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:15,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77dbee88[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:15,475 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:15,475 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:15,476 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:15,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:15,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:15,477 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:15,477 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:15,477 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:15,477 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,477 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:15,479 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,479 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:15,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:15,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:15,479 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:15,479 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:15,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:15,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:15,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:15,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:15,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:15,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:15,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:15,482 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:15,482 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:15,482 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:15,482 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:15,482 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:15,487 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:15,487 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:15,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:15,487 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:15,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:15,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:15,488 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:15,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,489 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:15,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,491 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,491 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:15,491 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:15,491 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,491 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:15,491 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:15,491 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:15,491 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:15,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:15,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:15,491 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:15,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:15,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:15,491 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:15,491 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:15,491 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:15,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:15,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:15,592 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:15,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:15,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:15,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:15,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:15,595 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:15,595 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:15,595 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:15,595 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:15,596 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:15,596 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:15,597 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,597 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:15,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:15,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:15,597 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,597 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:15,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:15,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:15,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:15,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:15,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:15,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:15,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:15,599 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:15,599 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:15,599 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:15,600 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:15,600 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:15,600 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:15,600 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:15,600 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-30 16:56:15,600 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:15,600 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:15,600 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:15,600 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:15,601 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:15,601 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:15,614 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033024a202429691447c822a7770e1e5209b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033024a202429691447c822a7770e1e5209b 2023-03-30 16:56:15,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:15,744 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:15,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:15,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195435784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:15,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195435784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:15,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195435889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:15,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:15,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195435889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,019 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=339, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/f851f6cd017a40e9bb6abd7271c0cb55 2023-03-30 16:56:16,028 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=339 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/f09589d004a145e5b2b64e8a9f539444 2023-03-30 16:56:16,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:16,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195436092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:16,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195436093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:16,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195436396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:16,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195436398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,438 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=339 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/3afa5edb487a4573bbe6ca23000d9c96 2023-03-30 16:56:16,442 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/f851f6cd017a40e9bb6abd7271c0cb55 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/f851f6cd017a40e9bb6abd7271c0cb55 2023-03-30 16:56:16,445 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/f851f6cd017a40e9bb6abd7271c0cb55, entries=150, sequenceid=339, filesize=25.4 K 2023-03-30 16:56:16,446 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/f09589d004a145e5b2b64e8a9f539444 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/f09589d004a145e5b2b64e8a9f539444 2023-03-30 16:56:16,449 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/f09589d004a145e5b2b64e8a9f539444, entries=150, sequenceid=339, filesize=11.9 K 2023-03-30 16:56:16,450 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/3afa5edb487a4573bbe6ca23000d9c96 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/3afa5edb487a4573bbe6ca23000d9c96 2023-03-30 16:56:16,453 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/3afa5edb487a4573bbe6ca23000d9c96, entries=150, sequenceid=339, filesize=11.9 K 2023-03-30 16:56:16,454 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 61fee4b7e4fa8ab90f65e22421bf6487 in 854ms, sequenceid=339, compaction requested=true 2023-03-30 16:56:16,454 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:16,454 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:16,454 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:16,454 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:16,454 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,454 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:16,454 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:16,456 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:16,456 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:16,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:16,456 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:16,456 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:16,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:16,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:16,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:16,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:16,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:16,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@590fb6ac[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:16,458 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:16,458 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:16,459 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:16,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:16,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:16,459 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:16,459 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:16,459 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:16,459 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,459 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:16,460 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:16,460 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:16,460 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,460 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:16,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:16,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:16,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:16,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:16,463 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:16,463 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:16,463 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:16,463 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:16,463 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:16,466 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:16,466 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:16,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:16,466 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:16,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:16,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:16,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:16,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:16,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:16,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:16,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:16,467 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:16,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:16,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:16,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:16,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,468 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:16,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:16,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,471 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,471 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:16,471 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:16,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:16,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:16,471 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:16,471 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:16,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:16,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:16,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:16,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:16,471 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:16,472 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,471 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:16,472 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:16,472 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:16,472 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:16,472 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:16,572 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:16,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:16,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:16,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:16,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:16,575 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:16,575 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:16,575 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:16,575 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:16,576 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:16,576 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:16,577 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:16,577 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:16,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:16,577 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,577 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:16,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:16,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:16,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:16,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:16,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:16,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:16,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:16,579 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:16,579 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:16,579 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:16,580 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:16,580 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:16,580 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:16,580 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:16,580 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 16:56:16,581 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:16,581 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:16,581 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:16,581 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:16,581 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:16,581 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:16,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:16,905 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:16,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680195376924, max timestamp=1680195376948Num uniques 859; 2023-03-30 16:56:16,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:56:16,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680195376924, max timestamp=1680195376948Num uniques 859; 2023-03-30 16:56:16,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:56:16,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680195376924, max timestamp=1680195376948Num uniques 859; 2023-03-30 16:56:16,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-30 16:56:16,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:16,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195436977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:16,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195436978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:16,997 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303305d3ee80456544a1483f1584558106d37 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303305d3ee80456544a1483f1584558106d37 2023-03-30 16:56:17,004 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=350, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/69c96ee468324507af7abe652de0ebc7 2023-03-30 16:56:17,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:17,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195437081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:17,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195437081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:17,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195437284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:17,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195437285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,413 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/aa2c018b92b445019c424bc4886e32c5 2023-03-30 16:56:17,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:17,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195437589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:17,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195437590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,823 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/c7864a6a6d0f4945bf3580793541a6a5 2023-03-30 16:56:17,828 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/69c96ee468324507af7abe652de0ebc7 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/69c96ee468324507af7abe652de0ebc7 2023-03-30 16:56:17,832 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/69c96ee468324507af7abe652de0ebc7, entries=150, sequenceid=350, filesize=25.4 K 2023-03-30 16:56:17,832 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/aa2c018b92b445019c424bc4886e32c5 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/aa2c018b92b445019c424bc4886e32c5 2023-03-30 16:56:17,836 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/aa2c018b92b445019c424bc4886e32c5, entries=150, sequenceid=350, filesize=11.9 K 2023-03-30 16:56:17,837 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/c7864a6a6d0f4945bf3580793541a6a5 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/c7864a6a6d0f4945bf3580793541a6a5 2023-03-30 16:56:17,840 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/c7864a6a6d0f4945bf3580793541a6a5, entries=150, sequenceid=350, filesize=11.9 K 2023-03-30 16:56:17,841 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 61fee4b7e4fa8ab90f65e22421bf6487 in 1261ms, sequenceid=350, compaction requested=true 2023-03-30 16:56:17,841 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:17,841 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:17,841 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:17,843 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:17,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:17,843 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:17,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:17,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f730889[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:17,845 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:17,845 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:17,847 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:17,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:17,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:17,847 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:17,847 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:17,847 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,847 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:17,847 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:17,849 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:17,849 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:17,849 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,849 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:17,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:17,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:17,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:17,852 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:17,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:17,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:17,852 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:17,852 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:17,853 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:17,853 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:17,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:17,854 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:17,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:17,854 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:17,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:17,855 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:17,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:17,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:17,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,858 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,858 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:17,859 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:17,859 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,859 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:17,859 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:17,859 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:17,858 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:17,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:17,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:17,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:17,859 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:17,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:17,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:17,859 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:17,859 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:17,859 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:17,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:17,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:17,959 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:17,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:17,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:17,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:17,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:17,963 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:17,963 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:17,963 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:17,963 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:17,964 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:17,964 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:17,969 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:17,969 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:17,969 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:17,969 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:17,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:17,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:17,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:17,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:17,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:17,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:17,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:17,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:17,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:17,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:17,971 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:17,971 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:17,971 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:17,971 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-30 16:56:17,971 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:17,972 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:17,972 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:17,972 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:17,972 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:17,972 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:18,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:18,097 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:18,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:18,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195438111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:18,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:18,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195438111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:18,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:18,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195438215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:18,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:18,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195438215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:18,385 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033062fc95d63cdc44d0b443ed5300bc0ac1 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033062fc95d63cdc44d0b443ed5300bc0ac1 2023-03-30 16:56:18,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:18,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195438419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:18,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:18,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195438420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:18,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:18,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195438724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:18,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:18,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195438727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:18,790 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=385, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/f1b6554ecbac4f10ade0f63354e4a16a 2023-03-30 16:56:18,801 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/f4f0887c0f604632a1af983882168525 2023-03-30 16:56:19,225 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/1a61e6b5293c4d388cba52f3769cbc63 2023-03-30 16:56:19,229 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/f1b6554ecbac4f10ade0f63354e4a16a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/f1b6554ecbac4f10ade0f63354e4a16a 2023-03-30 16:56:19,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:19,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195439229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:19,233 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/f1b6554ecbac4f10ade0f63354e4a16a, entries=150, sequenceid=385, filesize=25.4 K 2023-03-30 16:56:19,233 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/f4f0887c0f604632a1af983882168525 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/f4f0887c0f604632a1af983882168525 2023-03-30 16:56:19,237 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/f4f0887c0f604632a1af983882168525, entries=150, sequenceid=385, filesize=11.9 K 2023-03-30 16:56:19,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:19,238 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/1a61e6b5293c4d388cba52f3769cbc63 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/1a61e6b5293c4d388cba52f3769cbc63 2023-03-30 16:56:19,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195439234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:19,241 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/1a61e6b5293c4d388cba52f3769cbc63, entries=150, sequenceid=385, filesize=11.9 K 2023-03-30 16:56:19,241 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 61fee4b7e4fa8ab90f65e22421bf6487 in 1270ms, sequenceid=385, compaction requested=true 2023-03-30 16:56:19,241 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:19,241 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:19,241 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:19,242 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:19,242 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:19,242 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:19,242 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:19,243 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:19,243 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:19,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:19,244 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:19,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:19,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:19,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:19,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ba905bc[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:19,245 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:19,245 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:19,246 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:19,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:19,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:19,247 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:19,247 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:19,247 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:19,247 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:19,247 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:19,248 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:19,248 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:19,248 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:19,248 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:19,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:19,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:19,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:19,251 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:19,251 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:19,252 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:19,252 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:19,252 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:19,256 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:19,256 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:19,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:19,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:19,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:19,257 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:19,258 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:19,259 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:19,265 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:19,265 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:19,265 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:19,265 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:19,265 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:19,265 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:19,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:19,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:19,265 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:19,266 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:19,265 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:19,265 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:19,266 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:19,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:19,266 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:19,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:19,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:19,366 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:19,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:19,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:19,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:19,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:19,370 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:19,370 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:19,370 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:19,370 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:19,371 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:19,371 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:19,372 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:19,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:19,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:19,372 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:19,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:19,373 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:19,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:19,373 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:19,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:19,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:19,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:19,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:19,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:19,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:19,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:19,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:19,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:19,374 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:19,374 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:19,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:19,374 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-30 16:56:19,375 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:19,375 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:19,375 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:19,375 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:19,375 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:19,375 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:19,787 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330c7c345c315c7430eb43c5ad6d4d9b379 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330c7c345c315c7430eb43c5ad6d4d9b379 2023-03-30 16:56:19,793 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=396, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/5bd4aa51628345ecb770459a701fbb54 2023-03-30 16:56:19,805 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/75aaca89cf6c43ac92606b945143efd9 2023-03-30 16:56:20,216 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/522f3ab84f274563b6fa1d15d37ed734 2023-03-30 16:56:20,220 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/5bd4aa51628345ecb770459a701fbb54 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/5bd4aa51628345ecb770459a701fbb54 2023-03-30 16:56:20,224 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/5bd4aa51628345ecb770459a701fbb54, entries=100, sequenceid=396, filesize=18.6 K 2023-03-30 16:56:20,225 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/75aaca89cf6c43ac92606b945143efd9 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/75aaca89cf6c43ac92606b945143efd9 2023-03-30 16:56:20,229 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/75aaca89cf6c43ac92606b945143efd9, entries=100, sequenceid=396, filesize=9.5 K 2023-03-30 16:56:20,230 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/522f3ab84f274563b6fa1d15d37ed734 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/522f3ab84f274563b6fa1d15d37ed734 2023-03-30 16:56:20,234 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/522f3ab84f274563b6fa1d15d37ed734, entries=100, sequenceid=396, filesize=9.5 K 2023-03-30 16:56:20,235 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 61fee4b7e4fa8ab90f65e22421bf6487 in 861ms, sequenceid=396, compaction requested=true 2023-03-30 16:56:20,235 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:20,235 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:20,235 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:20,235 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:20,235 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,235 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:20,235 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:20,243 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,243 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:20,243 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,243 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:20,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73ae0fef[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:20,248 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:20,249 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,249 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,250 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:20,250 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:20,250 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:20,250 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,250 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:20,251 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:20,251 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:20,251 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:20,251 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:20,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:20,254 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:20,254 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:20,254 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:20,254 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:20,254 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:20,256 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,256 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,256 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:20,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,256 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:20,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,258 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:20,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,262 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,262 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,262 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:20,262 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:20,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:20,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:20,262 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,262 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,262 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:20,262 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:20,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:20,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:20,262 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:20,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:20,263 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:20,263 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:20,262 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:20,284 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-30 16:56:20,285 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:20,285 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:20,285 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:20,285 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:20,285 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:20,286 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:20,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680195380300, max timestamp=1680195380324Num uniques 859; 2023-03-30 16:56:20,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 16:56:20,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680195380300, max timestamp=1680195380324Num uniques 859; 2023-03-30 16:56:20,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680195380300, max timestamp=1680195380324Num uniques 859; 2023-03-30 16:56:20,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 16:56:20,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-30 16:56:20,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:20,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195440353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:20,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195440355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,363 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:20,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:20,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:20,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:20,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:20,366 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:20,366 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:20,366 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:20,366 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:20,366 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,367 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,368 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:20,368 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:20,368 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,368 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:20,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:20,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:20,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:20,369 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:20,369 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:20,369 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:20,369 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:20,369 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:20,369 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:20,370 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:20,370 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:20,370 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:20,370 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:20,370 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:20,370 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,370 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:20,370 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:20,371 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,371 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:20,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:20,371 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:20,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:20,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:20,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:20,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:20,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57307de0[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:20,373 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:20,373 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,374 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:20,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:20,374 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:20,375 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,375 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:20,378 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,378 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:20,378 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:20,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:20,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:20,378 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:20,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:20,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:20,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:20,380 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:20,380 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:20,380 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:20,380 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:20,380 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:20,382 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,382 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:20,382 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:20,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:20,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:20,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:20,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:20,382 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:20,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,383 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:20,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,385 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,385 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,385 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,386 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,386 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,386 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,385 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:20,386 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:20,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:20,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:20,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:20,386 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:20,386 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:20,386 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:20,386 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:20,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:20,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:20,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195440457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:20,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195440461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,486 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:20,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:20,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:20,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:20,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:20,490 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:20,490 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:20,490 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:20,490 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:20,491 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,491 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,492 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:20,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:20,492 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,492 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,493 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:20,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:20,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:20,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:20,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:20,493 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:20,493 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:20,493 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:20,494 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:20,494 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:20,494 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:20,494 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:20,494 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:20,494 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:20,494 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:20,494 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:20,494 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,494 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:20,494 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:20,496 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,496 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:20,497 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:20,497 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:20,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:20,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@306010cf[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:20,498 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:20,498 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,499 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:20,500 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,500 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:20,501 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,501 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:20,501 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:20,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:20,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:20,501 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:20,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:20,504 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:20,504 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:20,504 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:20,504 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:20,504 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:20,506 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,506 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:20,506 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:20,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,506 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,509 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,514 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,514 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,514 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:20,515 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:20,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:20,515 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,515 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,515 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,515 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,515 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:20,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:20,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:20,515 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:20,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:20,515 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:20,515 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:20,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:20,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:20,615 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:20,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:20,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:20,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:20,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:20,620 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:20,620 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:20,620 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:20,620 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:20,624 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,624 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,625 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:20,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:20,626 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,626 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,626 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:20,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,627 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:20,627 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:20,627 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:20,627 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:20,627 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:20,627 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:20,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:20,628 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:20,628 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:20,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:20,628 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:20,628 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:20,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:20,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:20,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:20,628 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:20,630 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,630 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:20,630 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,630 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:20,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:20,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:20,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:20,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:20,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23f7679a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:20,632 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:20,632 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,633 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,633 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:20,633 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:20,633 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:20,633 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,633 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:20,635 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:20,635 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:20,635 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:20,636 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:20,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:20,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:20,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:20,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:20,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:20,638 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:20,638 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:20,638 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:20,638 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:20,638 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:20,640 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:20,640 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,640 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:20,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:20,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:20,641 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,642 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,645 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,645 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:20,645 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:20,645 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:20,645 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,645 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,645 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,645 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:20,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:20,645 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:20,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:20,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:20,645 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,646 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:20,646 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:20,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:20,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195440661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:20,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195440665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,698 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303302e7f62a34a6e44fb8ef44fab382360f2 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303302e7f62a34a6e44fb8ef44fab382360f2 2023-03-30 16:56:20,746 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:20,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:20,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:20,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:20,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:20,750 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:20,750 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:20,750 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:20,750 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:20,750 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,750 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,752 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:20,752 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:20,752 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,752 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:20,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:20,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:20,755 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:20,755 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:20,755 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:20,755 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:20,755 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:20,755 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:20,755 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:20,755 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:20,755 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:20,756 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:20,756 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:20,756 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,756 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:20,756 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:20,758 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,758 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:20,758 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,758 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:20,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:20,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20a5d57c[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:20,760 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:20,760 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,762 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,762 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:20,762 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:20,762 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,762 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:20,762 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:20,763 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,763 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:20,763 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:20,763 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:20,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:20,766 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:20,766 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:20,766 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:20,766 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:20,766 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:20,768 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,768 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:20,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,768 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:20,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:20,768 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:20,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:20,769 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,773 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:20,773 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:20,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:20,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:20,773 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:20,773 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,773 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,773 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:20,773 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:20,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:20,773 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,773 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:20,773 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:20,773 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,774 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,873 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:20,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:20,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:20,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:20,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:20,877 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:20,877 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:20,877 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:20,877 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:20,878 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,878 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,879 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:20,879 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:20,879 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,879 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:20,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:20,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:20,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:20,881 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:20,881 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:20,881 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:20,881 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:20,883 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,883 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:20,883 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,883 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:20,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:20,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@742e73f5[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:20,885 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:20,885 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,887 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,887 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:20,887 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:20,887 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:20,887 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,887 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:20,890 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:20,890 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:20,890 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,890 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:20,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:20,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:20,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:20,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:20,893 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:20,893 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:20,893 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:20,893 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:20,893 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:20,894 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,895 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:20,895 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:20,895 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:20,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:20,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:20,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,897 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:20,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:20,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:20,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:20,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,899 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:20,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:20,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:20,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:20,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:20,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:20,899 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:20,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:20,900 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:20,900 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:20,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:20,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:20,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:20,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195440967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:20,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:20,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195440970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,000 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:21,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:21,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:21,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:21,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:21,004 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:21,004 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:21,004 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:21,004 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:21,004 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,004 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,005 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:21,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:21,005 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,006 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,006 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:21,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:21,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:21,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:21,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:21,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:21,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:21,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:21,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:21,007 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:21,007 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:21,007 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:21,008 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:21,008 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:21,008 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:21,008 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:21,008 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,008 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:21,008 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:21,013 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,013 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,013 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,013 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:21,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:21,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4478f3d7[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:21,015 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:21,015 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,016 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,016 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:21,016 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,016 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:21,016 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:21,016 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:21,017 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,018 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:21,018 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:21,018 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:21,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:21,020 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:21,020 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:21,020 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:21,020 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:21,020 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:21,022 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,022 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,022 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:21,023 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,024 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,028 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,028 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:21,028 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:21,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:21,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:21,028 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:21,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:21,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:21,028 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:21,028 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,028 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:21,029 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:21,029 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,029 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,029 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,029 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:21,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:21,103 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=407, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/9be031cf7317466588785a4533fd6ae4 2023-03-30 16:56:21,129 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:21,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:21,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:21,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:21,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:21,132 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:21,132 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:21,132 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:21,132 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:21,132 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,132 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,133 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,133 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:21,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:21,134 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,134 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:21,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:21,135 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:21,135 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:21,135 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:21,135 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:21,135 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:21,135 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:21,135 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:21,135 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:21,135 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:21,135 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:21,135 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:21,135 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:21,135 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,135 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:21,135 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:21,137 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,137 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,137 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,137 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:21,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cab9775[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:21,139 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:21,139 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,140 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:21,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:21,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:21,140 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,140 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:21,142 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,142 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:21,142 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:21,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,142 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:21,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:21,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:21,144 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:21,145 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:21,145 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:21,145 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:21,145 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:21,146 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,146 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,146 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:21,147 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,148 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,150 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:21,150 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:21,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:21,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:21,151 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:21,151 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,151 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:21,151 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,151 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:21,151 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,151 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:21,151 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,151 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,151 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:21,251 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:21,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:21,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:21,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:21,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:21,255 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:21,255 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:21,255 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:21,255 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:21,255 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,255 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,256 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:21,256 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:21,257 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,257 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:21,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:21,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:21,258 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:21,258 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:21,258 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:21,258 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:21,258 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:21,258 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:21,258 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:21,258 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:21,258 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:21,259 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:21,259 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:21,259 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,259 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:21,259 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:21,260 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,260 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,260 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,261 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:21,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52abb7a9[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:21,262 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:21,262 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,263 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,263 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:21,263 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:21,263 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:21,263 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,264 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:21,271 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,271 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,271 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:21,271 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:21,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:21,273 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:21,273 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:21,273 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:21,273 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:21,274 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:21,275 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,275 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:21,275 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,276 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,277 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,280 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,280 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,280 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,280 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:21,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:21,280 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:21,280 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:21,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:21,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:21,280 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:21,280 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:21,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:21,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:21,280 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:21,280 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,281 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,281 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,380 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:21,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:21,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:21,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:21,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:21,384 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:21,384 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:21,384 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:21,384 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:21,384 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,385 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,386 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:21,386 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,386 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,386 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:21,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:21,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:21,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:21,387 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:21,387 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:21,387 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:21,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:21,388 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:21,388 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:21,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:21,388 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:21,388 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:21,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:21,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:21,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:21,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:21,392 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,392 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,392 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,392 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:21,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@551dc7d2[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:21,394 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:21,394 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,400 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,400 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:21,400 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,400 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:21,400 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:21,400 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:21,401 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,402 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:21,402 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,402 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:21,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:21,404 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:21,404 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:21,404 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:21,404 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:21,404 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:21,407 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,407 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,407 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:21,408 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,409 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,412 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:21,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:21,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:21,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:21,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:21,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:21,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:21,413 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:21,413 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:21,412 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:21,413 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:21,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195441472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:21,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195441475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,512 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/e868bc7e3aac43ea801b930070eb3354 2023-03-30 16:56:21,513 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:21,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:21,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:21,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:21,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:21,517 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:21,517 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:21,517 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:21,517 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:21,517 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,517 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,518 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,518 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:21,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:21,520 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,520 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:21,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:21,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:21,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:21,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:21,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:21,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:21,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:21,521 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:21,521 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:21,521 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:21,521 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:21,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:21,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:21,521 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,522 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:21,522 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:21,523 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,524 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,524 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:21,524 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/c41b840e224f417d96c8a8670b1263c4 2023-03-30 16:56:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:21,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56f870ed[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:21,526 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:21,526 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,528 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:21,529 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,529 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:21,529 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/9be031cf7317466588785a4533fd6ae4 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/9be031cf7317466588785a4533fd6ae4 2023-03-30 16:56:21,530 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,530 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:21,530 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:21,530 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:21,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:21,533 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:21,533 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:21,533 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:21,533 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:21,533 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:21,534 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,535 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/9be031cf7317466588785a4533fd6ae4, entries=250, sequenceid=407, filesize=39.2 K 2023-03-30 16:56:21,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,535 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,535 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:21,536 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/e868bc7e3aac43ea801b930070eb3354 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e868bc7e3aac43ea801b930070eb3354 2023-03-30 16:56:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,537 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,539 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:21,539 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:21,539 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:21,539 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e868bc7e3aac43ea801b930070eb3354, entries=150, sequenceid=407, filesize=11.9 K 2023-03-30 16:56:21,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:21,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:21,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:21,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:21,539 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:21,539 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,540 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:21,540 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,540 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:21,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:21,540 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,540 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,540 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,540 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,540 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/c41b840e224f417d96c8a8670b1263c4 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/c41b840e224f417d96c8a8670b1263c4 2023-03-30 16:56:21,543 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/c41b840e224f417d96c8a8670b1263c4, entries=150, sequenceid=407, filesize=11.9 K 2023-03-30 16:56:21,544 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 61fee4b7e4fa8ab90f65e22421bf6487 in 1260ms, sequenceid=407, compaction requested=true 2023-03-30 16:56:21,544 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:21,544 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:56:21,544 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-30 16:56:21,544 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-30 16:56:21,544 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:56:21,544 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:56:21,546 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 215800 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-30 16:56:21,546 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 61fee4b7e4fa8ab90f65e22421bf6487/A is initiating minor compaction (all files) 2023-03-30 16:56:21,546 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61fee4b7e4fa8ab90f65e22421bf6487/A in TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:21,546 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/52d7384513784d59af083d9480a80108, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/7be553d9370d4a3a9fe5f687c483588c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/a9e4b5edfe7a4d3482e42d201752822e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/f851f6cd017a40e9bb6abd7271c0cb55, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/69c96ee468324507af7abe652de0ebc7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/f1b6554ecbac4f10ade0f63354e4a16a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/5bd4aa51628345ecb770459a701fbb54, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/9be031cf7317466588785a4533fd6ae4] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp, totalSize=210.7 K 2023-03-30 16:56:21,547 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 52d7384513784d59af083d9480a80108, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680195372233 2023-03-30 16:56:21,547 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95440 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-30 16:56:21,547 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 61fee4b7e4fa8ab90f65e22421bf6487/B is initiating minor compaction (all files) 2023-03-30 16:56:21,547 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61fee4b7e4fa8ab90f65e22421bf6487/B in TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:21,547 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/820b7d3d7a92476cbb35b38f630146ac, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/cfa2c71eb1c8491382fb339066b01f78, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/115839d6bdf64374bc56d36f1902edf4, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/f09589d004a145e5b2b64e8a9f539444, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/aa2c018b92b445019c424bc4886e32c5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/f4f0887c0f604632a1af983882168525, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/75aaca89cf6c43ac92606b945143efd9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e868bc7e3aac43ea801b930070eb3354] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp, totalSize=93.2 K 2023-03-30 16:56:21,547 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 7be553d9370d4a3a9fe5f687c483588c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680195372314 2023-03-30 16:56:21,547 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 820b7d3d7a92476cbb35b38f630146ac, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680195372233 2023-03-30 16:56:21,547 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting a9e4b5edfe7a4d3482e42d201752822e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1680195374460 2023-03-30 16:56:21,548 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting cfa2c71eb1c8491382fb339066b01f78, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680195372314 2023-03-30 16:56:21,548 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting f851f6cd017a40e9bb6abd7271c0cb55, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1680195375120 2023-03-30 16:56:21,548 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 115839d6bdf64374bc56d36f1902edf4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1680195374460 2023-03-30 16:56:21,548 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 69c96ee468324507af7abe652de0ebc7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1680195375752 2023-03-30 16:56:21,548 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting f09589d004a145e5b2b64e8a9f539444, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1680195375120 2023-03-30 16:56:21,548 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting f1b6554ecbac4f10ade0f63354e4a16a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680195376969 2023-03-30 16:56:21,549 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting aa2c018b92b445019c424bc4886e32c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1680195375752 2023-03-30 16:56:21,549 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 5bd4aa51628345ecb770459a701fbb54, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1680195378109 2023-03-30 16:56:21,549 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting f4f0887c0f604632a1af983882168525, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680195376969 2023-03-30 16:56:21,549 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 9be031cf7317466588785a4533fd6ae4, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680195380257 2023-03-30 16:56:21,549 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 75aaca89cf6c43ac92606b945143efd9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1680195378109 2023-03-30 16:56:21,550 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting e868bc7e3aac43ea801b930070eb3354, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680195380279 2023-03-30 16:56:21,566 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 61fee4b7e4fa8ab90f65e22421bf6487#B#compaction#531 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:56:21,578 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61fee4b7e4fa8ab90f65e22421bf6487#A#compaction#530 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:56:21,601 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/92a42f3c6f554bd9ba5351f1e0f7a71a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/92a42f3c6f554bd9ba5351f1e0f7a71a 2023-03-30 16:56:21,605 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 61fee4b7e4fa8ab90f65e22421bf6487/A of 61fee4b7e4fa8ab90f65e22421bf6487 into 92a42f3c6f554bd9ba5351f1e0f7a71a(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:56:21,605 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:21,605 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., storeName=61fee4b7e4fa8ab90f65e22421bf6487/A, priority=8, startTime=1680195381544; duration=0sec 2023-03-30 16:56:21,605 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:56:21,605 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-30 16:56:21,607 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95440 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-30 16:56:21,607 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 61fee4b7e4fa8ab90f65e22421bf6487/C is initiating minor compaction (all files) 2023-03-30 16:56:21,607 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61fee4b7e4fa8ab90f65e22421bf6487/C in TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:21,607 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/9a72d3cb776340e7ac05a4a88456ebba, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/d7605159ca9a40229b1476c4260caa8a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/30ea69b733f84379b298af2bbc201a49, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/3afa5edb487a4573bbe6ca23000d9c96, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/c7864a6a6d0f4945bf3580793541a6a5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/1a61e6b5293c4d388cba52f3769cbc63, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/522f3ab84f274563b6fa1d15d37ed734, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/c41b840e224f417d96c8a8670b1263c4] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp, totalSize=93.2 K 2023-03-30 16:56:21,607 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 9a72d3cb776340e7ac05a4a88456ebba, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680195372233 2023-03-30 16:56:21,608 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting d7605159ca9a40229b1476c4260caa8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680195372314 2023-03-30 16:56:21,608 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 30ea69b733f84379b298af2bbc201a49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1680195374460 2023-03-30 16:56:21,608 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 3afa5edb487a4573bbe6ca23000d9c96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1680195375120 2023-03-30 16:56:21,608 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting c7864a6a6d0f4945bf3580793541a6a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1680195375752 2023-03-30 16:56:21,609 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 1a61e6b5293c4d388cba52f3769cbc63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680195376969 2023-03-30 16:56:21,609 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 522f3ab84f274563b6fa1d15d37ed734, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1680195378109 2023-03-30 16:56:21,609 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting c41b840e224f417d96c8a8670b1263c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680195380279 2023-03-30 16:56:21,619 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61fee4b7e4fa8ab90f65e22421bf6487#C#compaction#532 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:56:21,640 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:21,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:21,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:21,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:21,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:21,643 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:21,643 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:21,643 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:21,643 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:21,643 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,643 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,649 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,649 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:21,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:21,649 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,649 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:21,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:21,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:21,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:21,650 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:21,650 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:21,650 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:21,651 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:21,651 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:21,651 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-30 16:56:21,651 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:21,651 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:21,651 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:21,651 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:21,651 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:21,651 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:21,668 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033033912eda45214394835d4bf3a45e7605 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033033912eda45214394835d4bf3a45e7605 2023-03-30 16:56:21,673 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=442, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/25fed1962ba34764932709831ca6cc27 2023-03-30 16:56:21,681 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/8fede9cd57d64023be6eed9777a1f4d2 2023-03-30 16:56:21,689 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/b4818a192b014bd787b12575688d39c7 2023-03-30 16:56:21,693 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/25fed1962ba34764932709831ca6cc27 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/25fed1962ba34764932709831ca6cc27 2023-03-30 16:56:21,696 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/25fed1962ba34764932709831ca6cc27, entries=150, sequenceid=442, filesize=25.4 K 2023-03-30 16:56:21,697 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/8fede9cd57d64023be6eed9777a1f4d2 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/8fede9cd57d64023be6eed9777a1f4d2 2023-03-30 16:56:21,700 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/8fede9cd57d64023be6eed9777a1f4d2, entries=150, sequenceid=442, filesize=11.9 K 2023-03-30 16:56:21,701 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/b4818a192b014bd787b12575688d39c7 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/b4818a192b014bd787b12575688d39c7 2023-03-30 16:56:21,704 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/b4818a192b014bd787b12575688d39c7, entries=150, sequenceid=442, filesize=11.9 K 2023-03-30 16:56:21,705 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=0 B/0 for 61fee4b7e4fa8ab90f65e22421bf6487 in 54ms, sequenceid=442, compaction requested=false 2023-03-30 16:56:21,705 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:21,705 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:21,705 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:21,707 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,707 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,707 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,707 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:21,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56684578[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:21,709 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:21,709 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,710 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,710 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:21,710 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:21,710 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,710 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:21,710 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:21,712 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,712 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:21,712 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,712 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:21,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:21,714 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:21,714 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:21,714 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:21,714 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:21,714 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:21,717 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,717 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,717 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:21,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:21,718 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,719 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,721 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,721 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:21,721 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:21,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:21,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:21,721 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:21,721 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:21,721 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:21,721 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:21,721 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:21,722 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,722 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:21,722 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,722 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,822 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:21,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:21,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:21,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:21,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:21,825 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:21,825 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:21,825 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:21,825 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:21,825 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,825 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,827 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:21,827 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:21,827 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,827 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:21,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:21,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:21,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:21,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:21,828 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:21,829 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:21,829 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:21,829 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:21,829 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:21,829 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:21,829 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:21,829 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:21,829 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:21,829 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,829 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:21,829 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:21,830 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,830 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,831 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,831 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:21,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59f1a6e3[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:21,832 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:21,832 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,834 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,834 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:21,834 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:21,834 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:21,834 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,835 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:21,839 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:21,839 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,839 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:21,839 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:21,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:21,841 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:21,841 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:21,841 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:21,841 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:21,841 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:21,843 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,843 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,843 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:21,844 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,845 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,849 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,849 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:21,849 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,849 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:21,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:21,849 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:21,849 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:21,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:21,849 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,849 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,849 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,849 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,849 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:21,849 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:21,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:21,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:21,950 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:21,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:21,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:21,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:21,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:21,972 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:21,972 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:21,972 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:21,972 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:21,972 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,972 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,974 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:21,974 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:21,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:21,974 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,974 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:21,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:21,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,975 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:21,975 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:21,975 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:21,975 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:21,975 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:21,976 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:21,976 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:21,976 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:21,976 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:21,976 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:21,976 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:21,976 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:21,976 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:21,976 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,976 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:21,976 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:21,978 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,978 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,978 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,978 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:21,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62e9c958[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:21,980 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:21,980 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,981 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,981 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:21,981 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:21,981 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,982 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:21,981 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:21,984 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/3ecb5d65ae1e439cb2e7975c0e6fe0a3 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/3ecb5d65ae1e439cb2e7975c0e6fe0a3 2023-03-30 16:56:21,986 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:21,986 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,986 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:21,986 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,988 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 61fee4b7e4fa8ab90f65e22421bf6487/B of 61fee4b7e4fa8ab90f65e22421bf6487 into 3ecb5d65ae1e439cb2e7975c0e6fe0a3(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:56:21,988 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:21,988 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., storeName=61fee4b7e4fa8ab90f65e22421bf6487/B, priority=8, startTime=1680195381544; duration=0sec 2023-03-30 16:56:21,989 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:56:21,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:21,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:21,989 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:21,989 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:21,989 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:21,989 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:21,989 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:21,992 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:21,993 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,993 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:21,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:21,993 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,994 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,996 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:21,996 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:21,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:21,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:21,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:21,996 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:21,996 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:21,996 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,997 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:21,997 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:21,997 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:21,997 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,997 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:21,997 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:21,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:21,997 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,027 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/88ea41864afc41dba23ea413d20ffe23 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/88ea41864afc41dba23ea413d20ffe23 2023-03-30 16:56:22,031 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 61fee4b7e4fa8ab90f65e22421bf6487/C of 61fee4b7e4fa8ab90f65e22421bf6487 into 88ea41864afc41dba23ea413d20ffe23(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:56:22,031 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:22,031 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., storeName=61fee4b7e4fa8ab90f65e22421bf6487/C, priority=8, startTime=1680195381544; duration=0sec 2023-03-30 16:56:22,032 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:56:22,097 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:22,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:22,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:22,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:22,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:22,105 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:22,105 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:22,105 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:22,105 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:22,105 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,105 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:22,107 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:22,107 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:22,107 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,107 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:22,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:22,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:22,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:22,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:22,113 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:22,113 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:22,113 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:22,113 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:22,113 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:22,113 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:22,113 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:22,114 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:22,114 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:22,114 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:22,114 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:22,114 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,114 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:22,114 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:22,116 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,116 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:22,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:22,117 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,117 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:22,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:22,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:22,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:22,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:22,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:22,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7783b2f0[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:22,118 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:22,118 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,119 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,120 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:22,120 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:22,120 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:22,120 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,120 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:22,121 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:22,121 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:22,121 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,121 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:22,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:22,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:22,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:22,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:22,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:22,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:22,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:22,124 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:22,124 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:22,124 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:22,124 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:22,124 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:22,126 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,126 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,126 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:22,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:22,127 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:22,128 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,136 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:22,136 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:22,136 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:22,136 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:22,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:22,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:22,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:22,136 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:22,136 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,136 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:22,136 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:22,136 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:22,136 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,137 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,137 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:22,237 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:22,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:22,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:22,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:22,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:22,240 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:22,240 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:22,240 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:22,240 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:22,245 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,245 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:22,246 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:22,246 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:22,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:22,247 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:22,247 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:22,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:22,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:22,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:22,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:22,248 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:22,248 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:22,249 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:22,249 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:22,249 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:22,249 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:22,251 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,251 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:22,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:22,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:22,252 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:22,252 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:22,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:22,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63d46afd[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:22,256 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:22,256 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,258 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,258 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:22,258 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,258 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:22,258 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:22,258 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:22,262 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:22,262 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:22,262 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:22,262 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:22,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:22,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:22,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:22,265 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:22,265 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:22,265 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:22,265 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:22,265 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:22,266 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,266 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,266 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:22,267 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:22,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:22,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:22,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:22,268 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,273 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,273 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:22,273 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:22,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:22,273 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:22,273 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:22,273 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:22,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:22,273 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:22,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:22,274 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,274 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,274 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:22,274 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:22,274 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:22,374 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:22,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:22,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:22,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:22,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:22,378 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:22,378 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:22,378 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:22,378 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:22,378 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,379 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:22,383 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,383 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:22,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:22,384 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,384 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:22,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:22,385 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:22,385 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:22,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:22,385 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:22,385 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:22,385 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:22,385 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:22,385 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:22,385 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:22,385 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:22,386 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:22,386 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:22,386 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:22,386 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:22,386 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,386 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:22,386 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:22,387 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,387 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:22,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:22,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:22,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:22,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37460eda[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:22,389 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:22,389 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,390 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,390 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:22,390 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:22,391 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:22,391 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,391 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:22,392 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:22,392 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,392 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:22,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:22,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:22,392 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:22,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:22,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:22,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:22,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:22,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:22,395 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:22,395 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:22,395 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:22,395 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:22,395 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:22,398 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,398 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,398 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:22,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:22,398 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:22,399 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:22,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,402 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,402 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:22,402 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:22,402 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:22,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:22,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:22,402 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,402 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,402 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:22,402 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,402 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:22,402 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:22,402 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:22,402 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:22,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:22,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:22,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:22,502 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:22,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:22,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:22,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:22,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:22,508 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:22,508 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:22,508 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:22,508 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:22,513 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,513 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:22,514 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,514 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:22,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:22,515 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,515 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:22,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:22,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:22,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:22,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:22,516 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:22,516 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:22,516 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:22,517 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:22,517 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:22,517 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:22,517 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:22,517 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-30 16:56:22,517 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:22,517 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:22,517 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:22,517 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:22,517 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:22,518 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:22,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:22,521 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:22,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195382539, max timestamp=1680195382550Num uniques 859; 2023-03-30 16:56:22,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:56:22,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195382539, max timestamp=1680195382550Num uniques 859; 2023-03-30 16:56:22,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:56:22,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680195382539, max timestamp=1680195382550Num uniques 859; 2023-03-30 16:56:22,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-30 16:56:22,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680195382570, max timestamp=1680195382579Num uniques 1018; 2023-03-30 16:56:22,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-30 16:56:22,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680195382570, max timestamp=1680195382579Num uniques 1018; 2023-03-30 16:56:22,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-30 16:56:22,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680195382570, max timestamp=1680195382579Num uniques 1018; 2023-03-30 16:56:22,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-30 16:56:22,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:22,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195442610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680195382605, max timestamp=1680195382613Num uniques 1077; 2023-03-30 16:56:22,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-30 16:56:22,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680195382605, max timestamp=1680195382613Num uniques 1077; 2023-03-30 16:56:22,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-30 16:56:22,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680195382605, max timestamp=1680195382613Num uniques 1077; 2023-03-30 16:56:22,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-30 16:56:22,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680195382650, max timestamp=1680195382669Num uniques 1036; 2023-03-30 16:56:22,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680195382650, max timestamp=1680195382669Num uniques 1036; 2023-03-30 16:56:22,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-30 16:56:22,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680195382650, max timestamp=1680195382669Num uniques 1036; 2023-03-30 16:56:22,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-30 16:56:22,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-30 16:56:22,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680195382707, max timestamp=1680195382727Num uniques 1045; 2023-03-30 16:56:22,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-30 16:56:22,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680195382707, max timestamp=1680195382727Num uniques 1045; 2023-03-30 16:56:22,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-30 16:56:22,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680195382707, max timestamp=1680195382727Num uniques 1045; 2023-03-30 16:56:22,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-30 16:56:22,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:22,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195442757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680195382749, max timestamp=1680195382759Num uniques 1054; 2023-03-30 16:56:22,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-30 16:56:22,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680195382749, max timestamp=1680195382759Num uniques 1054; 2023-03-30 16:56:22,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-30 16:56:22,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680195382749, max timestamp=1680195382759Num uniques 1054; 2023-03-30 16:56:22,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-30 16:56:22,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680195382784, max timestamp=1680195382817Num uniques 1013; 2023-03-30 16:56:22,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680195382784, max timestamp=1680195382817Num uniques 1013; 2023-03-30 16:56:22,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680195382784, max timestamp=1680195382817Num uniques 1013; 2023-03-30 16:56:22,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-30 16:56:22,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-30 16:56:22,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-30 16:56:22,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680195382863, max timestamp=1680195382874Num uniques 1022; 2023-03-30 16:56:22,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-03-30 16:56:22,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680195382863, max timestamp=1680195382874Num uniques 1022; 2023-03-30 16:56:22,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680195382863, max timestamp=1680195382874Num uniques 1022; 2023-03-30 16:56:22,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-03-30 16:56:22,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-03-30 16:56:22,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:22,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195442906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:22,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680195382901, max timestamp=1680195382908Num uniques 1081; 2023-03-30 16:56:22,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680195382901, max timestamp=1680195382908Num uniques 1081; 2023-03-30 16:56:22,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-30 16:56:22,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680195382901, max timestamp=1680195382908Num uniques 1081; 2023-03-30 16:56:22,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-30 16:56:22,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-30 16:56:22,934 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023033095222a99cf9341d6ba7620c6a3da4796 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033095222a99cf9341d6ba7620c6a3da4796 2023-03-30 16:56:22,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680195382955, max timestamp=1680195382968Num uniques 1040; 2023-03-30 16:56:22,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-03-30 16:56:22,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680195382955, max timestamp=1680195382968Num uniques 1040; 2023-03-30 16:56:22,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680195382955, max timestamp=1680195382968Num uniques 1040; 2023-03-30 16:56:22,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-03-30 16:56:22,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-03-30 16:56:23,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680195383009, max timestamp=1680195383018Num uniques 1049; 2023-03-30 16:56:23,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-30 16:56:23,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680195383009, max timestamp=1680195383018Num uniques 1049; 2023-03-30 16:56:23,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680195383009, max timestamp=1680195383018Num uniques 1049; 2023-03-30 16:56:23,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-30 16:56:23,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-30 16:56:23,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680195383041, max timestamp=1680195383056Num uniques 1058; 2023-03-30 16:56:23,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-03-30 16:56:23,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680195383041, max timestamp=1680195383056Num uniques 1058; 2023-03-30 16:56:23,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680195383041, max timestamp=1680195383056Num uniques 1058; 2023-03-30 16:56:23,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-03-30 16:56:23,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-03-30 16:56:23,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680195383074, max timestamp=1680195383091Num uniques 1067; 2023-03-30 16:56:23,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680195383074, max timestamp=1680195383091Num uniques 1067; 2023-03-30 16:56:23,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-03-30 16:56:23,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680195383074, max timestamp=1680195383091Num uniques 1017; 2023-03-30 16:56:23,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-03-30 16:56:23,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-03-30 16:56:23,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680195383116, max timestamp=1680195383122Num uniques 1026; 2023-03-30 16:56:23,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-30 16:56:23,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680195383116, max timestamp=1680195383122Num uniques 1026; 2023-03-30 16:56:23,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680195383116, max timestamp=1680195383122Num uniques 1026; 2023-03-30 16:56:23,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-30 16:56:23,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-30 16:56:23,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680195383134, max timestamp=1680195383153Num uniques 1035; 2023-03-30 16:56:23,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-03-30 16:56:23,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680195383134, max timestamp=1680195383153Num uniques 1035; 2023-03-30 16:56:23,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-03-30 16:56:23,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680195383134, max timestamp=1680195383153Num uniques 1035; 2023-03-30 16:56:23,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-03-30 16:56:23,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680195383176, max timestamp=1680195383185Num uniques 1094; 2023-03-30 16:56:23,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680195383176, max timestamp=1680195383185Num uniques 1094; 2023-03-30 16:56:23,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-03-30 16:56:23,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-03-30 16:56:23,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680195383176, max timestamp=1680195383185Num uniques 1094; 2023-03-30 16:56:23,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-03-30 16:56:23,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9288, totalHeapSize=21352, min timestamp=1680195383212, max timestamp=1680195383223Num uniques 1103; 2023-03-30 16:56:23,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50508, new segment data size=9288, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-03-30 16:56:23,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9288, totalHeapSize=21352, min timestamp=1680195383212, max timestamp=1680195383223Num uniques 1103; 2023-03-30 16:56:23,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9288, totalHeapSize=21352, min timestamp=1680195383212, max timestamp=1680195383223Num uniques 1103; 2023-03-30 16:56:23,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50508, new segment data size=9288, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-03-30 16:56:23,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50508, new segment data size=9288, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-03-30 16:56:23,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:23,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 419 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195443258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680195383253, max timestamp=1680195383260Num uniques 1062; 2023-03-30 16:56:23,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-03-30 16:56:23,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680195383253, max timestamp=1680195383260Num uniques 1062; 2023-03-30 16:56:23,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680195383253, max timestamp=1680195383260Num uniques 1062; 2023-03-30 16:56:23,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-03-30 16:56:23,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-03-30 16:56:23,340 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=457, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/cc5de8ce6cf14d7aa65f7f219e8c23be 2023-03-30 16:56:23,349 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/95b063ef5c894b7cb2a8351c309b621e 2023-03-30 16:56:23,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680195383323, max timestamp=1680195383351Num uniques 1021; 2023-03-30 16:56:23,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-03-30 16:56:23,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680195383323, max timestamp=1680195383351Num uniques 1021; 2023-03-30 16:56:23,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680195383323, max timestamp=1680195383351Num uniques 1021; 2023-03-30 16:56:23,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-03-30 16:56:23,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-03-30 16:56:23,361 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=457 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/074bdddb163341c99c455691224729db 2023-03-30 16:56:23,365 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/cc5de8ce6cf14d7aa65f7f219e8c23be as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/cc5de8ce6cf14d7aa65f7f219e8c23be 2023-03-30 16:56:23,370 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/cc5de8ce6cf14d7aa65f7f219e8c23be, entries=250, sequenceid=457, filesize=39.2 K 2023-03-30 16:56:23,370 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/95b063ef5c894b7cb2a8351c309b621e as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/95b063ef5c894b7cb2a8351c309b621e 2023-03-30 16:56:23,374 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/95b063ef5c894b7cb2a8351c309b621e, entries=150, sequenceid=457, filesize=11.9 K 2023-03-30 16:56:23,374 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/074bdddb163341c99c455691224729db as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/074bdddb163341c99c455691224729db 2023-03-30 16:56:23,379 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/074bdddb163341c99c455691224729db, entries=150, sequenceid=457, filesize=11.9 K 2023-03-30 16:56:23,379 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=100.63 KB/103050 for 61fee4b7e4fa8ab90f65e22421bf6487 in 862ms, sequenceid=457, compaction requested=true 2023-03-30 16:56:23,379 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:23,380 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:23,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:23,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:23,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:23,380 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:23,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:23,381 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,381 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=107.34 KB heapSize=304.17 KB 2023-03-30 16:56:23,382 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:23,382 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:23,382 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:23,382 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:23,382 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:23,382 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-30 16:56:23,381 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:23,382 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,382 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:23,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:23,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b49b263[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:23,385 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:23,385 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,387 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:23,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:23,388 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:23,388 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,388 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:23,389 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:23,390 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:23,390 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:23,390 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:23,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:23,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:23,393 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:23,393 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:23,393 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:23,393 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:23,393 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:23,394 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,394 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:23,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,395 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:23,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:23,395 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:23,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,396 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:23,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,398 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,398 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,398 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,399 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:23,399 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:23,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:23,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:23,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:23,399 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,399 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,399 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,399 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:23,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:23,399 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:23,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:23,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:23,399 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:23,399 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:23,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:23,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 475 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195443404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:23,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 431 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195443405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,499 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:23,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:23,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:23,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:23,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:23,503 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:23,503 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:23,503 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:23,503 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:23,503 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,503 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,505 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:23,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:23,505 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,506 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,506 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:23,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:23,507 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:23,507 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:23,507 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:23,507 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:23,507 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:23,507 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:23,507 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:23,507 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:23,507 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:23,507 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:23,508 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:23,508 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:23,508 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:23,508 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,508 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:23,508 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:23,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:23,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 477 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195443508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:23,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 433 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195443509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,509 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:23,510 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:23,510 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,510 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:23,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:23,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71b797c8[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:23,511 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:23,511 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,512 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,512 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:23,512 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:23,512 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:23,513 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,513 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:23,514 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,514 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:23,514 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:23,514 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:23,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:23,517 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:23,517 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:23,517 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:23,517 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:23,517 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:23,522 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,522 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:23,522 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:23,522 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:23,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,524 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:23,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,526 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,526 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:23,526 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:23,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:23,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:23,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:23,526 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:23,526 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,527 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:23,527 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,527 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,527 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:23,527 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:23,527 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,527 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:23,627 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:23,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:23,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:23,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:23,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:23,631 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:23,631 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:23,631 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:23,631 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:23,631 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,631 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,632 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:23,632 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:23,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:23,633 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,633 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:23,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:23,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:23,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:23,634 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:23,634 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:23,634 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:23,634 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:23,634 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:23,634 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:23,634 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:23,634 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:23,635 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:23,635 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:23,635 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:23,635 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,635 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:23,635 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:23,637 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,637 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:23,637 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,637 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:23,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31748881[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:23,639 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:23,639 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,641 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,641 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:23,641 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:23,641 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:23,641 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,641 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:23,642 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,642 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:23,642 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,642 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:23,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:23,646 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:23,646 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:23,646 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:23,646 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:23,646 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:23,648 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,648 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,648 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:23,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:23,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:23,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,648 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:23,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,650 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:23,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,653 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,653 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,653 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:23,654 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:23,654 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:23,653 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,654 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:23,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:23,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:23,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:23,654 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,654 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,654 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:23,654 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:23,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:23,654 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:23,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:23,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 479 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195443710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:23,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 435 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195443711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,754 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:23,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:23,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:23,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:23,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:23,757 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:23,758 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:23,758 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:23,758 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:23,758 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,758 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,759 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:23,760 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:23,760 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,760 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:23,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:23,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:23,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:23,761 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:23,761 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:23,761 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:23,761 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:23,761 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:23,761 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:23,761 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:23,761 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:23,761 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:23,762 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:23,762 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:23,762 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,762 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:23,762 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:23,763 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,763 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:23,763 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,763 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:23,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:23,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:23,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36c370ab[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:23,766 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:23,766 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,767 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,767 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:23,767 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:23,767 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:23,767 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,767 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:23,770 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,770 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:23,770 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:23,770 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:23,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:23,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:23,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:23,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:23,773 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:23,773 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:23,773 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:23,773 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:23,773 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:23,774 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,774 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:23,774 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:23,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,775 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:23,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:23,776 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,778 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,778 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,778 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,779 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,779 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:23,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:23,779 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:23,779 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:23,779 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,779 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,779 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:23,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:23,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:23,779 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:23,779 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:23,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:23,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:23,795 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330b3ca88a6216341ecaa27504a978c6b8d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330b3ca88a6216341ecaa27504a978c6b8d 2023-03-30 16:56:23,800 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=799, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/b0c4dd6752d84dabaf4b9dafec88e370 2023-03-30 16:56:23,809 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=799 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/ae2c4deb97ac46e3a61a7368fa5a568c 2023-03-30 16:56:23,879 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:23,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:23,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:23,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:23,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:23,883 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:23,883 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:23,883 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:23,883 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:23,883 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,883 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,884 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:23,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:23,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:23,884 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,885 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,885 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:23,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,885 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:23,885 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:23,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:23,886 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:23,886 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:23,886 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:23,886 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:23,886 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:23,886 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:23,886 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:23,886 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:23,886 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:23,886 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:23,886 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:23,886 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,886 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:23,886 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:23,888 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,888 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:23,889 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,889 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:23,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:23,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@604f06ba[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:23,890 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:23,890 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,891 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,891 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:23,891 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:23,891 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:23,892 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,892 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:23,893 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,893 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:23,893 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:23,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:23,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:23,893 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:23,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:23,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:23,895 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:23,895 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:23,895 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:23,895 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:23,895 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:23,897 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,897 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:23,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,897 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:23,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:23,897 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:23,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:23,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,898 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:23,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:23,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,901 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,901 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,901 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:23,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:23,901 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:23,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:23,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:23,901 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:23,902 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,902 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:23,902 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:23,901 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:23,902 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:23,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:23,902 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:23,902 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:23,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:24,002 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:24,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:24,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:24,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:24,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:24,006 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:24,006 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:24,006 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:24,006 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:24,006 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:24,006 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:24,010 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:24,010 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:24,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:24,010 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,011 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:24,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:24,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:24,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:24,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:24,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:24,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:24,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:24,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:24,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:24,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:24,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:24,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:24,012 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:24,012 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:24,012 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:24,012 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:24,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:24,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:24,012 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,013 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:24,013 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:24,014 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:24,014 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:24,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:24,014 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:24,014 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:24,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:24,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:24,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:24,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:24,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:24,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d0bd4ac[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:24,016 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:24,016 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:24,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:24,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 437 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195444014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:24,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 481 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195444014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,018 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:24,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:24,019 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:24,019 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:24,019 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:24,019 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,019 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:24,020 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,020 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:24,020 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:24,020 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:24,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:24,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:24,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:24,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:24,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:24,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:24,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:24,022 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:24,023 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:24,023 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:24,023 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:24,023 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:24,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:24,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:24,024 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:24,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:24,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:24,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:24,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:24,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:24,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:24,025 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,026 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,028 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:24,028 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:24,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:24,028 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:24,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:24,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:24,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:24,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:24,028 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:24,029 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:24,028 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:24,030 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:24,030 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:24,030 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,030 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:24,030 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:24,030 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:24,130 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:24,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:24,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:24,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:24,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:24,133 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:24,133 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:24,133 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:24,133 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:24,134 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:24,134 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:24,136 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,136 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:24,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:24,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:24,137 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,137 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:24,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:24,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:24,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:24,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:24,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:24,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:24,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:24,138 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:24,138 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:24,138 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:24,138 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:24,138 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:24,138 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:24,138 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:24,138 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:24,139 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:24,139 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:24,139 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:24,139 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,139 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:24,139 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:24,140 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:24,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:24,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:24,141 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:24,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:24,141 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:24,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:24,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:24,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:24,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:24,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@382e610c[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:24,143 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:24,143 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:24,144 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:24,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:24,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:24,144 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:24,144 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:24,144 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:24,144 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,144 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:24,146 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:24,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:24,146 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:24,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:24,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:24,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:24,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:24,148 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:24,148 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:24,148 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:24,148 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:24,148 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:24,150 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:24,151 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:24,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:24,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:24,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:24,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:24,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:24,151 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:24,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:24,151 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:24,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:24,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:24,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:24,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:24,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:24,152 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:24,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,155 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,155 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:24,155 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:24,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:24,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:24,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:24,155 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:24,155 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:24,155 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:24,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:24,155 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:24,155 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:24,155 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,156 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:24,156 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:24,156 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:24,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:24,219 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=799 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/9ad51516300742e08b91fc40da0545c8 2023-03-30 16:56:24,223 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/b0c4dd6752d84dabaf4b9dafec88e370 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/b0c4dd6752d84dabaf4b9dafec88e370 2023-03-30 16:56:24,227 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/b0c4dd6752d84dabaf4b9dafec88e370, entries=200, sequenceid=799, filesize=32.3 K 2023-03-30 16:56:24,227 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/ae2c4deb97ac46e3a61a7368fa5a568c as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/ae2c4deb97ac46e3a61a7368fa5a568c 2023-03-30 16:56:24,231 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/ae2c4deb97ac46e3a61a7368fa5a568c, entries=150, sequenceid=799, filesize=11.9 K 2023-03-30 16:56:24,232 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/9ad51516300742e08b91fc40da0545c8 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/9ad51516300742e08b91fc40da0545c8 2023-03-30 16:56:24,235 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/9ad51516300742e08b91fc40da0545c8, entries=150, sequenceid=799, filesize=11.9 K 2023-03-30 16:56:24,236 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~304.13 KB/311424, currentSize=73.80 KB/75570 for 61fee4b7e4fa8ab90f65e22421bf6487 in 855ms, sequenceid=799, compaction requested=true 2023-03-30 16:56:24,236 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:24,236 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:56:24,236 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:56:24,236 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:56:24,236 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:56:24,237 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:56:24,238 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 126126 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:56:24,238 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1912): 61fee4b7e4fa8ab90f65e22421bf6487/A is initiating minor compaction (all files) 2023-03-30 16:56:24,238 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61fee4b7e4fa8ab90f65e22421bf6487/A in TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:24,238 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/92a42f3c6f554bd9ba5351f1e0f7a71a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/25fed1962ba34764932709831ca6cc27, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/cc5de8ce6cf14d7aa65f7f219e8c23be, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/b0c4dd6752d84dabaf4b9dafec88e370] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp, totalSize=123.2 K 2023-03-30 16:56:24,238 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 92a42f3c6f554bd9ba5351f1e0f7a71a, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680195380279 2023-03-30 16:56:24,239 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting 25fed1962ba34764932709831ca6cc27, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680195380346 2023-03-30 16:56:24,239 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:56:24,239 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 61fee4b7e4fa8ab90f65e22421bf6487/B is initiating minor compaction (all files) 2023-03-30 16:56:24,239 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61fee4b7e4fa8ab90f65e22421bf6487/B in TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:24,239 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting cc5de8ce6cf14d7aa65f7f219e8c23be, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1680195382490 2023-03-30 16:56:24,239 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/3ecb5d65ae1e439cb2e7975c0e6fe0a3, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/8fede9cd57d64023be6eed9777a1f4d2, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/95b063ef5c894b7cb2a8351c309b621e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/ae2c4deb97ac46e3a61a7368fa5a568c] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp, totalSize=48.4 K 2023-03-30 16:56:24,240 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 3ecb5d65ae1e439cb2e7975c0e6fe0a3, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680195380279 2023-03-30 16:56:24,240 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] compactions.Compactor(207): Compacting b0c4dd6752d84dabaf4b9dafec88e370, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=799, earliestPutTs=1680195383369 2023-03-30 16:56:24,244 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 8fede9cd57d64023be6eed9777a1f4d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680195380346 2023-03-30 16:56:24,244 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 95b063ef5c894b7cb2a8351c309b621e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1680195382501 2023-03-30 16:56:24,244 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting ae2c4deb97ac46e3a61a7368fa5a568c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=799, earliestPutTs=1680195383369 2023-03-30 16:56:24,255 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:24,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:24,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:24,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:24,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:24,273 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 61fee4b7e4fa8ab90f65e22421bf6487#B#compaction#542 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:56:24,274 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:24,274 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:24,274 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:24,274 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:24,274 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:24,274 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:24,276 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,276 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:24,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:24,276 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,276 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:24,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:24,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:24,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:24,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:24,277 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61fee4b7e4fa8ab90f65e22421bf6487#A#compaction#543 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:56:24,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:24,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:24,284 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:24,284 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:24,284 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:24,288 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:24,288 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:24,288 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:24,288 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:24,288 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-30 16:56:24,289 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:24,289 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:24,289 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:24,289 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:24,289 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:24,289 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:24,312 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/76aae0909d214600913978d717b21708 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/76aae0909d214600913978d717b21708 2023-03-30 16:56:24,317 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61fee4b7e4fa8ab90f65e22421bf6487/B of 61fee4b7e4fa8ab90f65e22421bf6487 into 76aae0909d214600913978d717b21708(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:56:24,318 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:24,318 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., storeName=61fee4b7e4fa8ab90f65e22421bf6487/B, priority=12, startTime=1680195384236; duration=0sec 2023-03-30 16:56:24,318 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-30 16:56:24,318 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-30 16:56:24,319 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-30 16:56:24,319 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1912): 61fee4b7e4fa8ab90f65e22421bf6487/C is initiating minor compaction (all files) 2023-03-30 16:56:24,319 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61fee4b7e4fa8ab90f65e22421bf6487/C in TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:24,319 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/88ea41864afc41dba23ea413d20ffe23, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/b4818a192b014bd787b12575688d39c7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/074bdddb163341c99c455691224729db, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/9ad51516300742e08b91fc40da0545c8] into tmpdir=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp, totalSize=48.4 K 2023-03-30 16:56:24,319 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 88ea41864afc41dba23ea413d20ffe23, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680195380279 2023-03-30 16:56:24,320 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting b4818a192b014bd787b12575688d39c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680195380346 2023-03-30 16:56:24,320 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 074bdddb163341c99c455691224729db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=457, earliestPutTs=1680195382501 2023-03-30 16:56:24,320 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] compactions.Compactor(207): Compacting 9ad51516300742e08b91fc40da0545c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=799, earliestPutTs=1680195383369 2023-03-30 16:56:24,335 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] throttle.PressureAwareThroughputController(145): 61fee4b7e4fa8ab90f65e22421bf6487#C#compaction#545 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-30 16:56:24,348 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/9d1378c23c5d4a1389fc6c6fc18c2fc8 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/9d1378c23c5d4a1389fc6c6fc18c2fc8 2023-03-30 16:56:24,352 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61fee4b7e4fa8ab90f65e22421bf6487/C of 61fee4b7e4fa8ab90f65e22421bf6487 into 9d1378c23c5d4a1389fc6c6fc18c2fc8(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:56:24,352 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:24,352 INFO [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., storeName=61fee4b7e4fa8ab90f65e22421bf6487/C, priority=12, startTime=1680195384237; duration=0sec 2023-03-30 16:56:24,352 DEBUG [RS:0;jenkins-hbase4:45193-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:56:24,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(9158): Flush requested on 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:24,521 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. as already flushing 2023-03-30 16:56:24,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:24,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 490 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195444543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:24,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 447 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195444544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:24,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 492 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195444646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:24,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 449 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195444648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,732 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330b5057c74141248829c82e9d8b304e02f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330b5057c74141248829c82e9d8b304e02f 2023-03-30 16:56:24,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:24,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 494 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195444850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:24,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 451 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195444852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:24,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51126 deadline: 1680195444873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,877 DEBUG [Thread-3533] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18216 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:56:24,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:24,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51170 deadline: 1680195444878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,881 DEBUG [Thread-3537] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18221 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:56:24,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:24,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45193] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51108 deadline: 1680195444900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:24,903 DEBUG [Thread-3539] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18241 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., hostname=jenkins-hbase4.apache.org,45193,1680195214285, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-30 16:56:25,136 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=813, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/843450c9df1540c18f88a03c9d724e32 2023-03-30 16:56:25,137 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/0490cce75cd84cb98437737135c38aac as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/0490cce75cd84cb98437737135c38aac 2023-03-30 16:56:25,142 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61fee4b7e4fa8ab90f65e22421bf6487/A of 61fee4b7e4fa8ab90f65e22421bf6487 into 0490cce75cd84cb98437737135c38aac(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-30 16:56:25,142 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:25,142 INFO [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487., storeName=61fee4b7e4fa8ab90f65e22421bf6487/A, priority=12, startTime=1680195384236; duration=0sec 2023-03-30 16:56:25,142 DEBUG [RS:0;jenkins-hbase4:45193-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-30 16:56:25,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:25,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 496 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195445153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:25,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:25,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 453 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195445155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:25,546 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=813 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/a1aafdff263b4decaf07460c8c21e321 2023-03-30 16:56:25,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:25,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45193] ipc.CallRunner(144): callId: 498 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51160 deadline: 1680195445656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:25,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-30 16:56:25,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45193] ipc.CallRunner(144): callId: 455 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51118 deadline: 1680195445659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:25,956 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=813 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/b20b159daafa4f968e7b7c6157ba0734 2023-03-30 16:56:25,961 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/843450c9df1540c18f88a03c9d724e32 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/843450c9df1540c18f88a03c9d724e32 2023-03-30 16:56:25,964 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/843450c9df1540c18f88a03c9d724e32, entries=150, sequenceid=813, filesize=25.4 K 2023-03-30 16:56:25,965 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/a1aafdff263b4decaf07460c8c21e321 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/a1aafdff263b4decaf07460c8c21e321 2023-03-30 16:56:25,969 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/a1aafdff263b4decaf07460c8c21e321, entries=150, sequenceid=813, filesize=11.9 K 2023-03-30 16:56:25,969 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/b20b159daafa4f968e7b7c6157ba0734 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/b20b159daafa4f968e7b7c6157ba0734 2023-03-30 16:56:25,982 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/b20b159daafa4f968e7b7c6157ba0734, entries=150, sequenceid=813, filesize=11.9 K 2023-03-30 16:56:25,983 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 61fee4b7e4fa8ab90f65e22421bf6487 in 1695ms, sequenceid=813, compaction requested=false 2023-03-30 16:56:25,983 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:25,983 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:25,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:25,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:25,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:25,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:25,984 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:25,985 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:25,985 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:25,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:25,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:25,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:25,986 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:25,986 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:25,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:25,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:25,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:25,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:25,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:25,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:25,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1545088a[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:25,987 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:25,987 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:25,989 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:25,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:25,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:25,990 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:25,990 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:25,990 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:25,990 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:25,990 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:25,991 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:25,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:25,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:25,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:25,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:25,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:25,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:25,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:25,994 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:25,995 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:25,995 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:25,995 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:25,995 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:25,997 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:25,997 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:25,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:25,997 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:25,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:25,998 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:25,999 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:25,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,001 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,001 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:26,001 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:26,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:26,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:26,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:26,001 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:26,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:26,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:26,001 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,001 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,001 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,002 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,002 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,002 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:26,002 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,002 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:26,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:26,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:26,102 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:26,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:26,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:26,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:26,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:26,105 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:26,105 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:26,105 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:26,105 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:26,106 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,106 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,109 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:26,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:26,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:26,109 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,109 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,110 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:26,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:26,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:26,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:26,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:26,110 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:26,110 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:26,110 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:26,110 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:26,111 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:26,110 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:26,111 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:26,111 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-30 16:56:26,111 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:26,111 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:26,111 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:26,111 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:26,111 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:26,111 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:26,128 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330f5d098e959bf481b9b54b032825c4c38 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330f5d098e959bf481b9b54b032825c4c38 2023-03-30 16:56:26,136 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=834, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/7abc7a53bdf64cbb9733045dbde3dbaf 2023-03-30 16:56:26,145 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=834 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/28f3314c826347109278eb9a4f337c2d 2023-03-30 16:56:26,172 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=834 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/1303cd91ecfd412aa549ba5d993e52e6 2023-03-30 16:56:26,176 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/7abc7a53bdf64cbb9733045dbde3dbaf as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/7abc7a53bdf64cbb9733045dbde3dbaf 2023-03-30 16:56:26,179 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/7abc7a53bdf64cbb9733045dbde3dbaf, entries=150, sequenceid=834, filesize=25.4 K 2023-03-30 16:56:26,180 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/28f3314c826347109278eb9a4f337c2d as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/28f3314c826347109278eb9a4f337c2d 2023-03-30 16:56:26,183 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/28f3314c826347109278eb9a4f337c2d, entries=150, sequenceid=834, filesize=11.9 K 2023-03-30 16:56:26,183 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/1303cd91ecfd412aa549ba5d993e52e6 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/1303cd91ecfd412aa549ba5d993e52e6 2023-03-30 16:56:26,186 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/1303cd91ecfd412aa549ba5d993e52e6, entries=150, sequenceid=834, filesize=11.9 K 2023-03-30 16:56:26,187 INFO [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 61fee4b7e4fa8ab90f65e22421bf6487 in 76ms, sequenceid=834, compaction requested=true 2023-03-30 16:56:26,187 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:26,187 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:26,189 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:26,189 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:26,190 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,190 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:26,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:26,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:26,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:26,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d27a64e[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:26,191 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:26,191 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,194 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,194 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,194 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:26,194 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:26,194 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:26,194 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:26,197 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:26,197 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:26,197 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,197 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:26,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:26,200 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:26,200 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:26,200 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:26,200 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:26,200 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:26,201 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,202 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:26,202 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,202 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:26,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:26,203 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,205 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,205 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,205 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:26,205 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:26,205 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:26,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:26,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:26,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:26,205 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,206 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:26,206 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:26,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:26,205 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:26,206 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,206 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,206 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,306 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:26,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:26,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:26,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:26,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:26,312 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:26,312 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:26,312 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:26,312 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:26,313 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,313 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,314 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,314 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:26,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:26,314 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,314 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:26,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:26,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:26,316 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:26,316 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:26,316 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:26,316 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:26,316 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:26,316 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:26,316 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:26,317 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:26,317 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:26,317 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:26,317 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:26,317 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,317 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:26,317 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:26,319 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,319 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:26,320 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,320 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:26,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:26,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7efabdf2[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:26,322 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:26,322 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,323 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,323 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:26,323 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:26,323 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:26,323 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,323 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:26,324 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,324 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:26,324 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:26,324 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:26,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:26,327 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:26,327 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:26,327 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:26,327 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:26,327 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:26,329 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,329 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,329 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:26,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:26,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:26,330 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,331 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:26,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,334 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:26,334 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:26,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:26,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:26,334 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:26,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:26,334 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,335 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:26,335 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,335 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,335 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:26,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:26,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:26,335 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,335 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,335 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,335 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,435 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:26,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:26,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:26,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:26,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:26,440 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:26,440 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:26,440 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:26,440 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:26,440 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,440 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,441 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:26,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:26,442 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,442 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,442 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:26,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:26,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:26,443 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:26,443 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:26,443 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:26,443 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:26,443 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:26,444 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:26,444 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:26,444 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:26,444 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:26,446 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,446 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:26,446 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,446 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:26,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:26,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:26,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:26,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35979520[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:26,448 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:26,448 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,449 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,449 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:26,449 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:26,449 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:26,449 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,449 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:26,453 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,453 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:26,453 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:26,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:26,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:26,453 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:26,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:26,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:26,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:26,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:26,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:26,455 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:26,456 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:26,456 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:26,456 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:26,456 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:26,457 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,457 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:26,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,457 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:26,458 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:26,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:26,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,460 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:26,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,462 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:26,462 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:26,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:26,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:26,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:26,462 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:26,462 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,462 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,462 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,462 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,462 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,462 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,462 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:26,462 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:26,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:26,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:26,462 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:26,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:26,563 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:26,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:26,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-30 16:56:26,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-30 16:56:26,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-30 16:56:26,566 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-30 16:56:26,566 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:26,566 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-30 16:56:26,566 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-30 16:56:26,566 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,566 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,567 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,567 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:26,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:26,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:26,568 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,568 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-30 16:56:26,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-30 16:56:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-30 16:56:26,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-30 16:56:26,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-30 16:56:26,569 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-30 16:56:26,569 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-30 16:56:26,569 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-30 16:56:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-30 16:56:26,570 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:26,570 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. started... 2023-03-30 16:56:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-30 16:56:26,570 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:26,570 DEBUG [rs(jenkins-hbase4.apache.org,45193,1680195214285)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-30 16:56:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-30 16:56:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-30 16:56:26,570 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-30 16:56:26,572 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,572 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:26,573 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,573 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-30 16:56:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:26,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,45193,1680195214285' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-30 16:56:26,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8abca40[Count = 0] remaining members to acquire global barrier 2023-03-30 16:56:26,575 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-30 16:56:26,575 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,576 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,576 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-30 16:56:26,576 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-30 16:56:26,576 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,45193,1680195214285' in zk 2023-03-30 16:56:26,576 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,576 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-30 16:56:26,580 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,580 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-30 16:56:26,580 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:26,580 DEBUG [member: 'jenkins-hbase4.apache.org,45193,1680195214285' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-30 16:56:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,45193,1680195214285': 2023-03-30 16:56:26,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,45193,1680195214285' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-30 16:56:26,583 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-30 16:56:26,583 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-30 16:56:26,583 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-30 16:56:26,583 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-30 16:56:26,583 INFO [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-30 16:56:26,585 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,585 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,585 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:26,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-30 16:56:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-30 16:56:26,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-30 16:56:26,586 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-30 16:56:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,587 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-30 16:56:26,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-30 16:56:26,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,590 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,590 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,590 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-30 16:56:26,590 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:26,590 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-30 16:56:26,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-30 16:56:26,590 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-30 16:56:26,590 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-30 16:56:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-30 16:56:26,590 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-30 16:56:26,590 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-30 16:56:26,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-30 16:56:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-30 16:56:26,592 DEBUG [(jenkins-hbase4.apache.org,40565,1680195213071)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-30 16:56:26,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-30 16:56:26,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-30 16:56:26,594 DEBUG [Thread-3542] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x19c1e3ee to 127.0.0.1:58435 2023-03-30 16:56:26,594 DEBUG [Thread-3542] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:56:26,596 DEBUG [Thread-3541] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-30 16:56:26,596 DEBUG [Thread-3541] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-30 16:56:26,599 DEBUG [Thread-3546] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x17cf2e3f to 127.0.0.1:58435 2023-03-30 16:56:26,599 DEBUG [Thread-3546] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:56:26,600 DEBUG [Thread-3544] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x526f2221 to 127.0.0.1:58435 2023-03-30 16:56:26,600 DEBUG [Thread-3544] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:56:26,615 DEBUG [Thread-3550] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x26d7d0b5 to 127.0.0.1:58435 2023-03-30 16:56:26,615 DEBUG [Thread-3550] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:56:26,617 DEBUG [Thread-3548] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3420e54c to 127.0.0.1:58435 2023-03-30 16:56:26,617 DEBUG [Thread-3548] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:56:26,665 DEBUG [Thread-3535] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x089b6154 to 127.0.0.1:58435 2023-03-30 16:56:26,665 DEBUG [Thread-3531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54cb8b37 to 127.0.0.1:58435 2023-03-30 16:56:26,665 DEBUG [Thread-3535] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:56:26,666 DEBUG [Thread-3531] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:56:26,696 DEBUG [Thread-3541] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-30 16:56:26,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-30 16:56:34,912 DEBUG [Thread-3533] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x041dc777 to 127.0.0.1:58435 2023-03-30 16:56:34,912 DEBUG [Thread-3533] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:56:34,931 DEBUG [Thread-3537] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x01d207bb to 127.0.0.1:58435 2023-03-30 16:56:34,931 DEBUG [Thread-3537] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:56:34,988 DEBUG [Thread-3539] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5bcc330a to 127.0.0.1:58435 2023-03-30 16:56:34,988 DEBUG [Thread-3539] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:56:34,988 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-30 16:56:34,988 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 328 2023-03-30 16:56:34,988 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 11 2023-03-30 16:56:34,988 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 375 2023-03-30 16:56:34,988 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 11 2023-03-30 16:56:34,988 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(394): wrote 11 2023-03-30 16:56:34,988 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-30 16:56:34,988 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(398): read 3995 2023-03-30 16:56:34,988 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(398): read 3964 2023-03-30 16:56:34,988 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(398): read 3750 2023-03-30 16:56:34,988 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(398): read 3972 2023-03-30 16:56:34,988 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(398): read 3940 2023-03-30 16:56:34,988 INFO [Listener at localhost/38075] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-30 16:56:34,988 INFO [Listener at localhost/38075] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-30 16:56:34,988 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45d9b525 to 127.0.0.1:58435 2023-03-30 16:56:34,989 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:56:34,989 INFO [Listener at localhost/38075] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-30 16:56:34,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-30 16:56:34,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-30 16:56:34,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-30 16:56:34,993 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195394992"}]},"ts":"1680195394992"} 2023-03-30 16:56:34,993 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-30 16:56:34,995 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-30 16:56:34,995 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=61fee4b7e4fa8ab90f65e22421bf6487, UNASSIGN}] 2023-03-30 16:56:34,997 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=61fee4b7e4fa8ab90f65e22421bf6487, UNASSIGN 2023-03-30 16:56:34,997 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=61fee4b7e4fa8ab90f65e22421bf6487, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:34,997 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195394997"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680195394997"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680195394997"}]},"ts":"1680195394997"} 2023-03-30 16:56:34,998 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285}] 2023-03-30 16:56:35,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-30 16:56:35,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:35,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 61fee4b7e4fa8ab90f65e22421bf6487, disabling compactions & flushes 2023-03-30 16:56:35,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:35,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:35,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. after waiting 0 ms 2023-03-30 16:56:35,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:35,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 61fee4b7e4fa8ab90f65e22421bf6487 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-30 16:56:35,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=A 2023-03-30 16:56:35,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:35,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=B 2023-03-30 16:56:35,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:35,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61fee4b7e4fa8ab90f65e22421bf6487, store=C 2023-03-30 16:56:35,151 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-30 16:56:35,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-30 16:56:35,561 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230330b1c1734bd61b4f7993463e432d3bc845 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330b1c1734bd61b4f7993463e432d3bc845 2023-03-30 16:56:35,573 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-30 16:56:35,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-30 16:56:35,965 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=845, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/492caee1451849b79fb5554741f60807 2023-03-30 16:56:36,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-30 16:56:36,374 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=845 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/e9e3021461b347629c54d65c576f8466 2023-03-30 16:56:36,782 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=845 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/8f0d217a8552498580be89513aee19b0 2023-03-30 16:56:36,785 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/A/492caee1451849b79fb5554741f60807 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/492caee1451849b79fb5554741f60807 2023-03-30 16:56:36,787 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/492caee1451849b79fb5554741f60807, entries=150, sequenceid=845, filesize=25.4 K 2023-03-30 16:56:36,788 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/B/e9e3021461b347629c54d65c576f8466 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e9e3021461b347629c54d65c576f8466 2023-03-30 16:56:36,791 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e9e3021461b347629c54d65c576f8466, entries=150, sequenceid=845, filesize=11.9 K 2023-03-30 16:56:36,791 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/.tmp/C/8f0d217a8552498580be89513aee19b0 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/8f0d217a8552498580be89513aee19b0 2023-03-30 16:56:36,795 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/8f0d217a8552498580be89513aee19b0, entries=150, sequenceid=845, filesize=11.9 K 2023-03-30 16:56:36,795 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 61fee4b7e4fa8ab90f65e22421bf6487 in 1645ms, sequenceid=845, compaction requested=true 2023-03-30 16:56:36,816 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/8b1424aef8bf4700b26de1257ea6151b, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/f1e526a61ed545f194c355dcffae768e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/4eacc451ff45415494229c9c2877d4a6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/72f2ff5fa4384a449089a2008ca208eb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/0e63f8fe412743719ef82aa27c5a5cf3, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/e2918dba924b44f1a3116ea326ff5055, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/06855a59aab54694b50d0e025d3eb66a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/66c90f31481447da9907c4940fa17e10, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/3f82c4b9a84e4aeebdddaa5234b99bb2, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/adc2c50c0ded4264bfe95dabc31956c8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/6dd2535bad444dcdbca14654403cf167, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/0b9ea9f393724802ac6d1078e5b7b733, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/e7d2fa26e49543498550b8823efa1cfb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/39fce1ac08b94888a6013d543dae46eb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/2e84363205134912876b76efc895f657, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/6d960c15ed22434a9c58cc1ed30ebb81, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/52d7384513784d59af083d9480a80108, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/7be553d9370d4a3a9fe5f687c483588c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/a9e4b5edfe7a4d3482e42d201752822e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/f851f6cd017a40e9bb6abd7271c0cb55, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/69c96ee468324507af7abe652de0ebc7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/f1b6554ecbac4f10ade0f63354e4a16a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/5bd4aa51628345ecb770459a701fbb54, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/9be031cf7317466588785a4533fd6ae4, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/92a42f3c6f554bd9ba5351f1e0f7a71a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/25fed1962ba34764932709831ca6cc27, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/cc5de8ce6cf14d7aa65f7f219e8c23be, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/b0c4dd6752d84dabaf4b9dafec88e370] to archive 2023-03-30 16:56:36,817 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 16:56:36,819 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/8b1424aef8bf4700b26de1257ea6151b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/8b1424aef8bf4700b26de1257ea6151b 2023-03-30 16:56:36,820 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/f1e526a61ed545f194c355dcffae768e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/f1e526a61ed545f194c355dcffae768e 2023-03-30 16:56:36,821 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/4eacc451ff45415494229c9c2877d4a6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/4eacc451ff45415494229c9c2877d4a6 2023-03-30 16:56:36,821 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/72f2ff5fa4384a449089a2008ca208eb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/72f2ff5fa4384a449089a2008ca208eb 2023-03-30 16:56:36,822 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/0e63f8fe412743719ef82aa27c5a5cf3 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/0e63f8fe412743719ef82aa27c5a5cf3 2023-03-30 16:56:36,823 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/e2918dba924b44f1a3116ea326ff5055 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/e2918dba924b44f1a3116ea326ff5055 2023-03-30 16:56:36,824 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/06855a59aab54694b50d0e025d3eb66a to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/06855a59aab54694b50d0e025d3eb66a 2023-03-30 16:56:36,825 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/66c90f31481447da9907c4940fa17e10 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/66c90f31481447da9907c4940fa17e10 2023-03-30 16:56:36,826 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/3f82c4b9a84e4aeebdddaa5234b99bb2 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/3f82c4b9a84e4aeebdddaa5234b99bb2 2023-03-30 16:56:36,826 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/adc2c50c0ded4264bfe95dabc31956c8 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/adc2c50c0ded4264bfe95dabc31956c8 2023-03-30 16:56:36,827 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/6dd2535bad444dcdbca14654403cf167 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/6dd2535bad444dcdbca14654403cf167 2023-03-30 16:56:36,828 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/0b9ea9f393724802ac6d1078e5b7b733 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/0b9ea9f393724802ac6d1078e5b7b733 2023-03-30 16:56:36,829 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/e7d2fa26e49543498550b8823efa1cfb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/e7d2fa26e49543498550b8823efa1cfb 2023-03-30 16:56:36,830 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/39fce1ac08b94888a6013d543dae46eb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/39fce1ac08b94888a6013d543dae46eb 2023-03-30 16:56:36,831 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/2e84363205134912876b76efc895f657 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/2e84363205134912876b76efc895f657 2023-03-30 16:56:36,832 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/6d960c15ed22434a9c58cc1ed30ebb81 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/6d960c15ed22434a9c58cc1ed30ebb81 2023-03-30 16:56:36,833 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/52d7384513784d59af083d9480a80108 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/52d7384513784d59af083d9480a80108 2023-03-30 16:56:36,834 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/7be553d9370d4a3a9fe5f687c483588c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/7be553d9370d4a3a9fe5f687c483588c 2023-03-30 16:56:36,835 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/a9e4b5edfe7a4d3482e42d201752822e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/a9e4b5edfe7a4d3482e42d201752822e 2023-03-30 16:56:36,836 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/f851f6cd017a40e9bb6abd7271c0cb55 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/f851f6cd017a40e9bb6abd7271c0cb55 2023-03-30 16:56:36,836 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/69c96ee468324507af7abe652de0ebc7 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/69c96ee468324507af7abe652de0ebc7 2023-03-30 16:56:36,837 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/f1b6554ecbac4f10ade0f63354e4a16a to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/f1b6554ecbac4f10ade0f63354e4a16a 2023-03-30 16:56:36,838 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/5bd4aa51628345ecb770459a701fbb54 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/5bd4aa51628345ecb770459a701fbb54 2023-03-30 16:56:36,839 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/9be031cf7317466588785a4533fd6ae4 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/9be031cf7317466588785a4533fd6ae4 2023-03-30 16:56:36,839 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/92a42f3c6f554bd9ba5351f1e0f7a71a to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/92a42f3c6f554bd9ba5351f1e0f7a71a 2023-03-30 16:56:36,840 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/25fed1962ba34764932709831ca6cc27 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/25fed1962ba34764932709831ca6cc27 2023-03-30 16:56:36,841 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/cc5de8ce6cf14d7aa65f7f219e8c23be to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/cc5de8ce6cf14d7aa65f7f219e8c23be 2023-03-30 16:56:36,841 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/b0c4dd6752d84dabaf4b9dafec88e370 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/b0c4dd6752d84dabaf4b9dafec88e370 2023-03-30 16:56:36,844 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e0406609c4e44f5a82c5c394b8b131b5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/351b383f591a4c05892f49d01db57783, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/44fc1250f04f4bd98c67248f9da50942, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/d45291ad441e4cfc8769557dc30c8c42, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/9882a5fed8554c5db253ff27f9f3cdf3, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/d545b4acac9845b594504810efbfd36d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/565691543d9d45a4b5255e500d72c1c9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/4cbafb90084a40b2a4d136b914653234, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/05a654eb9f6044a0baa1d7198eb97252, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/8237e19065f04e5aa2954199d6743ac8, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/c182b77a21c645faa3cd60e68f14e7a4, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/c753488bb5ac410b88f6235ef4cc95a3, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e4805990270d4f3f98bfb0c43c924108, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/826730381a8943358b13062147194b1d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/c0c25d0116d041f0955138c9c9638754, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/820b7d3d7a92476cbb35b38f630146ac, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/13e5399f81524f64ba0767404859a7aa, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/cfa2c71eb1c8491382fb339066b01f78, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/115839d6bdf64374bc56d36f1902edf4, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/f09589d004a145e5b2b64e8a9f539444, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/aa2c018b92b445019c424bc4886e32c5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/f4f0887c0f604632a1af983882168525, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/75aaca89cf6c43ac92606b945143efd9, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/3ecb5d65ae1e439cb2e7975c0e6fe0a3, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e868bc7e3aac43ea801b930070eb3354, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/8fede9cd57d64023be6eed9777a1f4d2, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/95b063ef5c894b7cb2a8351c309b621e, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/ae2c4deb97ac46e3a61a7368fa5a568c] to archive 2023-03-30 16:56:36,844 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 16:56:36,845 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e0406609c4e44f5a82c5c394b8b131b5 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e0406609c4e44f5a82c5c394b8b131b5 2023-03-30 16:56:36,846 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/351b383f591a4c05892f49d01db57783 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/351b383f591a4c05892f49d01db57783 2023-03-30 16:56:36,847 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/44fc1250f04f4bd98c67248f9da50942 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/44fc1250f04f4bd98c67248f9da50942 2023-03-30 16:56:36,848 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/d45291ad441e4cfc8769557dc30c8c42 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/d45291ad441e4cfc8769557dc30c8c42 2023-03-30 16:56:36,849 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/9882a5fed8554c5db253ff27f9f3cdf3 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/9882a5fed8554c5db253ff27f9f3cdf3 2023-03-30 16:56:36,849 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/d545b4acac9845b594504810efbfd36d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/d545b4acac9845b594504810efbfd36d 2023-03-30 16:56:36,850 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/565691543d9d45a4b5255e500d72c1c9 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/565691543d9d45a4b5255e500d72c1c9 2023-03-30 16:56:36,851 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/4cbafb90084a40b2a4d136b914653234 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/4cbafb90084a40b2a4d136b914653234 2023-03-30 16:56:36,852 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/05a654eb9f6044a0baa1d7198eb97252 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/05a654eb9f6044a0baa1d7198eb97252 2023-03-30 16:56:36,852 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/8237e19065f04e5aa2954199d6743ac8 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/8237e19065f04e5aa2954199d6743ac8 2023-03-30 16:56:36,853 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/c182b77a21c645faa3cd60e68f14e7a4 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/c182b77a21c645faa3cd60e68f14e7a4 2023-03-30 16:56:36,854 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/c753488bb5ac410b88f6235ef4cc95a3 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/c753488bb5ac410b88f6235ef4cc95a3 2023-03-30 16:56:36,854 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e4805990270d4f3f98bfb0c43c924108 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e4805990270d4f3f98bfb0c43c924108 2023-03-30 16:56:36,855 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/826730381a8943358b13062147194b1d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/826730381a8943358b13062147194b1d 2023-03-30 16:56:36,856 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/c0c25d0116d041f0955138c9c9638754 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/c0c25d0116d041f0955138c9c9638754 2023-03-30 16:56:36,857 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/820b7d3d7a92476cbb35b38f630146ac to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/820b7d3d7a92476cbb35b38f630146ac 2023-03-30 16:56:36,857 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/13e5399f81524f64ba0767404859a7aa to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/13e5399f81524f64ba0767404859a7aa 2023-03-30 16:56:36,858 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/cfa2c71eb1c8491382fb339066b01f78 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/cfa2c71eb1c8491382fb339066b01f78 2023-03-30 16:56:36,859 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/115839d6bdf64374bc56d36f1902edf4 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/115839d6bdf64374bc56d36f1902edf4 2023-03-30 16:56:36,860 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/f09589d004a145e5b2b64e8a9f539444 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/f09589d004a145e5b2b64e8a9f539444 2023-03-30 16:56:36,860 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/aa2c018b92b445019c424bc4886e32c5 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/aa2c018b92b445019c424bc4886e32c5 2023-03-30 16:56:36,861 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/f4f0887c0f604632a1af983882168525 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/f4f0887c0f604632a1af983882168525 2023-03-30 16:56:36,862 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/75aaca89cf6c43ac92606b945143efd9 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/75aaca89cf6c43ac92606b945143efd9 2023-03-30 16:56:36,863 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/3ecb5d65ae1e439cb2e7975c0e6fe0a3 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/3ecb5d65ae1e439cb2e7975c0e6fe0a3 2023-03-30 16:56:36,863 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e868bc7e3aac43ea801b930070eb3354 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e868bc7e3aac43ea801b930070eb3354 2023-03-30 16:56:36,864 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/8fede9cd57d64023be6eed9777a1f4d2 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/8fede9cd57d64023be6eed9777a1f4d2 2023-03-30 16:56:36,865 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/95b063ef5c894b7cb2a8351c309b621e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/95b063ef5c894b7cb2a8351c309b621e 2023-03-30 16:56:36,866 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/ae2c4deb97ac46e3a61a7368fa5a568c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/ae2c4deb97ac46e3a61a7368fa5a568c 2023-03-30 16:56:36,868 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/8eab36bff36e402b9599d72184d474cb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/b116ad30f0dc42a9ab5345884486380d, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/2dd16d7e958c4bb3b5119c22cff0130c, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/27a663c5e1d24dc68d27197f498c0015, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/f4f2dc3e955449109da3549eacdaa0eb, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/d868df44cf53422f9ebb81c6c20976be, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/30a37962eddc4037b9fef953b7c06132, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/4f253933dbba40e9b915944427cadec3, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/347ccba285484b8b80c6499df9f0e7f7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/68ef3477e71a42dfb00f5740ab51bdbf, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/dd97f1c25a5e43058b6a7e7d9c86422a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/c1dd091701cb4bff9cb6538d9727b3b0, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/64ead87b2e4d4778878b3f81e0c86adf, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/f907f74f05c641658fba360bf6fc66db, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/dbf8d35cdc86400f998f8d5b2db87da6, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/9a72d3cb776340e7ac05a4a88456ebba, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/9e6305984d0c4ffaa873b7900f950280, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/d7605159ca9a40229b1476c4260caa8a, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/30ea69b733f84379b298af2bbc201a49, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/3afa5edb487a4573bbe6ca23000d9c96, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/c7864a6a6d0f4945bf3580793541a6a5, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/1a61e6b5293c4d388cba52f3769cbc63, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/522f3ab84f274563b6fa1d15d37ed734, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/88ea41864afc41dba23ea413d20ffe23, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/c41b840e224f417d96c8a8670b1263c4, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/b4818a192b014bd787b12575688d39c7, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/074bdddb163341c99c455691224729db, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/9ad51516300742e08b91fc40da0545c8] to archive 2023-03-30 16:56:36,868 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-30 16:56:36,869 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/8eab36bff36e402b9599d72184d474cb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/8eab36bff36e402b9599d72184d474cb 2023-03-30 16:56:36,870 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/b116ad30f0dc42a9ab5345884486380d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/b116ad30f0dc42a9ab5345884486380d 2023-03-30 16:56:36,871 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/2dd16d7e958c4bb3b5119c22cff0130c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/2dd16d7e958c4bb3b5119c22cff0130c 2023-03-30 16:56:36,872 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/27a663c5e1d24dc68d27197f498c0015 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/27a663c5e1d24dc68d27197f498c0015 2023-03-30 16:56:36,873 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/f4f2dc3e955449109da3549eacdaa0eb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/f4f2dc3e955449109da3549eacdaa0eb 2023-03-30 16:56:36,873 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/d868df44cf53422f9ebb81c6c20976be to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/d868df44cf53422f9ebb81c6c20976be 2023-03-30 16:56:36,874 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/30a37962eddc4037b9fef953b7c06132 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/30a37962eddc4037b9fef953b7c06132 2023-03-30 16:56:36,875 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/4f253933dbba40e9b915944427cadec3 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/4f253933dbba40e9b915944427cadec3 2023-03-30 16:56:36,876 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/347ccba285484b8b80c6499df9f0e7f7 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/347ccba285484b8b80c6499df9f0e7f7 2023-03-30 16:56:36,877 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/68ef3477e71a42dfb00f5740ab51bdbf to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/68ef3477e71a42dfb00f5740ab51bdbf 2023-03-30 16:56:36,878 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/dd97f1c25a5e43058b6a7e7d9c86422a to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/dd97f1c25a5e43058b6a7e7d9c86422a 2023-03-30 16:56:36,879 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/c1dd091701cb4bff9cb6538d9727b3b0 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/c1dd091701cb4bff9cb6538d9727b3b0 2023-03-30 16:56:36,879 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/64ead87b2e4d4778878b3f81e0c86adf to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/64ead87b2e4d4778878b3f81e0c86adf 2023-03-30 16:56:36,880 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/f907f74f05c641658fba360bf6fc66db to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/f907f74f05c641658fba360bf6fc66db 2023-03-30 16:56:36,881 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/dbf8d35cdc86400f998f8d5b2db87da6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/dbf8d35cdc86400f998f8d5b2db87da6 2023-03-30 16:56:36,882 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/9a72d3cb776340e7ac05a4a88456ebba to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/9a72d3cb776340e7ac05a4a88456ebba 2023-03-30 16:56:36,883 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/9e6305984d0c4ffaa873b7900f950280 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/9e6305984d0c4ffaa873b7900f950280 2023-03-30 16:56:36,883 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/d7605159ca9a40229b1476c4260caa8a to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/d7605159ca9a40229b1476c4260caa8a 2023-03-30 16:56:36,884 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/30ea69b733f84379b298af2bbc201a49 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/30ea69b733f84379b298af2bbc201a49 2023-03-30 16:56:36,885 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/3afa5edb487a4573bbe6ca23000d9c96 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/3afa5edb487a4573bbe6ca23000d9c96 2023-03-30 16:56:36,886 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/c7864a6a6d0f4945bf3580793541a6a5 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/c7864a6a6d0f4945bf3580793541a6a5 2023-03-30 16:56:36,887 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/1a61e6b5293c4d388cba52f3769cbc63 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/1a61e6b5293c4d388cba52f3769cbc63 2023-03-30 16:56:36,888 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/522f3ab84f274563b6fa1d15d37ed734 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/522f3ab84f274563b6fa1d15d37ed734 2023-03-30 16:56:36,889 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/88ea41864afc41dba23ea413d20ffe23 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/88ea41864afc41dba23ea413d20ffe23 2023-03-30 16:56:36,890 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/c41b840e224f417d96c8a8670b1263c4 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/c41b840e224f417d96c8a8670b1263c4 2023-03-30 16:56:36,891 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/b4818a192b014bd787b12575688d39c7 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/b4818a192b014bd787b12575688d39c7 2023-03-30 16:56:36,892 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/074bdddb163341c99c455691224729db to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/074bdddb163341c99c455691224729db 2023-03-30 16:56:36,893 DEBUG [StoreCloser-TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/9ad51516300742e08b91fc40da0545c8 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/9ad51516300742e08b91fc40da0545c8 2023-03-30 16:56:36,896 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/recovered.edits/848.seqid, newMaxSeqId=848, maxSeqId=4 2023-03-30 16:56:36,898 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487. 2023-03-30 16:56:36,898 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 61fee4b7e4fa8ab90f65e22421bf6487: 2023-03-30 16:56:36,899 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:36,899 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=61fee4b7e4fa8ab90f65e22421bf6487, regionState=CLOSED 2023-03-30 16:56:36,899 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680195396899"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680195396899"}]},"ts":"1680195396899"} 2023-03-30 16:56:36,901 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-03-30 16:56:36,901 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 61fee4b7e4fa8ab90f65e22421bf6487, server=jenkins-hbase4.apache.org,45193,1680195214285 in 1.9020 sec 2023-03-30 16:56:36,902 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-03-30 16:56:36,902 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=61fee4b7e4fa8ab90f65e22421bf6487, UNASSIGN in 1.9060 sec 2023-03-30 16:56:36,903 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680195396903"}]},"ts":"1680195396903"} 2023-03-30 16:56:36,904 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-30 16:56:36,905 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-30 16:56:36,906 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9150 sec 2023-03-30 16:56:37,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-30 16:56:37,096 INFO [Listener at localhost/38075] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-03-30 16:56:37,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-30 16:56:37,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:56:37,099 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:56:37,100 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:56:37,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-30 16:56:37,103 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:37,105 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A, FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B, FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C, FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/recovered.edits] 2023-03-30 16:56:37,107 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/0490cce75cd84cb98437737135c38aac to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/0490cce75cd84cb98437737135c38aac 2023-03-30 16:56:37,109 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/492caee1451849b79fb5554741f60807 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/492caee1451849b79fb5554741f60807 2023-03-30 16:56:37,110 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/7abc7a53bdf64cbb9733045dbde3dbaf to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/7abc7a53bdf64cbb9733045dbde3dbaf 2023-03-30 16:56:37,111 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/843450c9df1540c18f88a03c9d724e32 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/A/843450c9df1540c18f88a03c9d724e32 2023-03-30 16:56:37,113 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/28f3314c826347109278eb9a4f337c2d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/28f3314c826347109278eb9a4f337c2d 2023-03-30 16:56:37,114 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/76aae0909d214600913978d717b21708 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/76aae0909d214600913978d717b21708 2023-03-30 16:56:37,115 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/a1aafdff263b4decaf07460c8c21e321 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/a1aafdff263b4decaf07460c8c21e321 2023-03-30 16:56:37,116 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e9e3021461b347629c54d65c576f8466 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/B/e9e3021461b347629c54d65c576f8466 2023-03-30 16:56:37,118 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/1303cd91ecfd412aa549ba5d993e52e6 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/1303cd91ecfd412aa549ba5d993e52e6 2023-03-30 16:56:37,119 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/8f0d217a8552498580be89513aee19b0 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/8f0d217a8552498580be89513aee19b0 2023-03-30 16:56:37,120 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/9d1378c23c5d4a1389fc6c6fc18c2fc8 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/9d1378c23c5d4a1389fc6c6fc18c2fc8 2023-03-30 16:56:37,121 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/b20b159daafa4f968e7b7c6157ba0734 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/C/b20b159daafa4f968e7b7c6157ba0734 2023-03-30 16:56:37,122 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/recovered.edits/848.seqid to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487/recovered.edits/848.seqid 2023-03-30 16:56:37,123 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/.tmp/data/default/TestAcidGuarantees/61fee4b7e4fa8ab90f65e22421bf6487 2023-03-30 16:56:37,123 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-30 16:56:37,123 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-30 16:56:37,124 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-30 16:56:37,126 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033021796a71e1204e39812886cf1eda723c to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033021796a71e1204e39812886cf1eda723c 2023-03-30 16:56:37,127 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033024a202429691447c822a7770e1e5209b to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033024a202429691447c822a7770e1e5209b 2023-03-30 16:56:37,128 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303302e7f62a34a6e44fb8ef44fab382360f2 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303302e7f62a34a6e44fb8ef44fab382360f2 2023-03-30 16:56:37,129 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033033912eda45214394835d4bf3a45e7605 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033033912eda45214394835d4bf3a45e7605 2023-03-30 16:56:37,130 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033039bbac195a8745d7b44c081c2ba83ea1 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033039bbac195a8745d7b44c081c2ba83ea1 2023-03-30 16:56:37,130 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330552783560fbe40409f1ee0663eba6e6d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330552783560fbe40409f1ee0663eba6e6d 2023-03-30 16:56:37,131 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330558e741e6832493c9e9d8bf0be9f46bb to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330558e741e6832493c9e9d8bf0be9f46bb 2023-03-30 16:56:37,132 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303305670d1f104834d688b515b9933745b91 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303305670d1f104834d688b515b9933745b91 2023-03-30 16:56:37,133 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033058bbcbbab420440180a7d7028d644966 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033058bbcbbab420440180a7d7028d644966 2023-03-30 16:56:37,134 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303305c46970c27ee421f9de329edb7411071 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303305c46970c27ee421f9de329edb7411071 2023-03-30 16:56:37,134 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303305c7627fc76164ed48516e440929ca421 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303305c7627fc76164ed48516e440929ca421 2023-03-30 16:56:37,135 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303305c7dbc2639a1442b890fdeb30bf0b27f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303305c7dbc2639a1442b890fdeb30bf0b27f 2023-03-30 16:56:37,136 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303305d3ee80456544a1483f1584558106d37 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303305d3ee80456544a1483f1584558106d37 2023-03-30 16:56:37,137 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033062fc95d63cdc44d0b443ed5300bc0ac1 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033062fc95d63cdc44d0b443ed5300bc0ac1 2023-03-30 16:56:37,138 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330646009feb7bf48fab0755c810ff8b4af to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330646009feb7bf48fab0755c810ff8b4af 2023-03-30 16:56:37,138 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303306dbe507cb33f4ef6b8a9ea6dede21401 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303306dbe507cb33f4ef6b8a9ea6dede21401 2023-03-30 16:56:37,139 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303308b9d18663af342ff93adb2761e0abc94 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303308b9d18663af342ff93adb2761e0abc94 2023-03-30 16:56:37,140 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033095222a99cf9341d6ba7620c6a3da4796 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023033095222a99cf9341d6ba7620c6a3da4796 2023-03-30 16:56:37,141 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330a675a3f9e0e84de9b01fbbd98a47de5e to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330a675a3f9e0e84de9b01fbbd98a47de5e 2023-03-30 16:56:37,142 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330b1c1734bd61b4f7993463e432d3bc845 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330b1c1734bd61b4f7993463e432d3bc845 2023-03-30 16:56:37,143 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330b3ca88a6216341ecaa27504a978c6b8d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330b3ca88a6216341ecaa27504a978c6b8d 2023-03-30 16:56:37,143 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330b5057c74141248829c82e9d8b304e02f to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330b5057c74141248829c82e9d8b304e02f 2023-03-30 16:56:37,144 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330c7c345c315c7430eb43c5ad6d4d9b379 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330c7c345c315c7430eb43c5ad6d4d9b379 2023-03-30 16:56:37,145 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d81d5f468f774fa482b9c98b88559e1d to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330d81d5f468f774fa482b9c98b88559e1d 2023-03-30 16:56:37,146 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330e55bbfb02b95433a8955e438e2772992 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330e55bbfb02b95433a8955e438e2772992 2023-03-30 16:56:37,147 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330f5d098e959bf481b9b54b032825c4c38 to hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230330f5d098e959bf481b9b54b032825c4c38 2023-03-30 16:56:37,147 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-30 16:56:37,149 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:56:37,150 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-30 16:56:37,151 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-30 16:56:37,152 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:56:37,152 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-30 16:56:37,152 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680195397152"}]},"ts":"9223372036854775807"} 2023-03-30 16:56:37,153 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-30 16:56:37,153 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 61fee4b7e4fa8ab90f65e22421bf6487, NAME => 'TestAcidGuarantees,,1680195364899.61fee4b7e4fa8ab90f65e22421bf6487.', STARTKEY => '', ENDKEY => ''}] 2023-03-30 16:56:37,153 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-30 16:56:37,153 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680195397153"}]},"ts":"9223372036854775807"} 2023-03-30 16:56:37,154 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-30 16:56:37,156 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-30 16:56:37,156 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 59 msec 2023-03-30 16:56:37,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40565] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-30 16:56:37,201 INFO [Listener at localhost/38075] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-03-30 16:56:37,212 INFO [Listener at localhost/38075] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 227), OpenFileDescriptor=529 (was 546), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=488 (was 415) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=4802 (was 4735) - AvailableMemoryMB LEAK? - 2023-03-30 16:56:37,212 INFO [Listener at localhost/38075] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-03-30 16:56:37,212 INFO [Listener at localhost/38075] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-30 16:56:37,213 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x610f60f0 to 127.0.0.1:58435 2023-03-30 16:56:37,213 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:56:37,213 DEBUG [Listener at localhost/38075] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-03-30 16:56:37,213 DEBUG [Listener at localhost/38075] util.JVMClusterUtil(257): Found active master hash=185435837, stopped=false 2023-03-30 16:56:37,213 INFO [Listener at localhost/38075] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,40565,1680195213071 2023-03-30 16:56:37,214 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-30 16:56:37,214 INFO [Listener at localhost/38075] procedure2.ProcedureExecutor(629): Stopping 2023-03-30 16:56:37,214 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-30 16:56:37,214 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-30 16:56:37,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-30 16:56:37,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-30 16:56:37,215 DEBUG [Listener at localhost/38075] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2869d048 to 127.0.0.1:58435 2023-03-30 16:56:37,215 DEBUG [Listener at localhost/38075] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:56:37,215 INFO [Listener at localhost/38075] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,45193,1680195214285' ***** 2023-03-30 16:56:37,215 INFO [Listener at localhost/38075] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-30 16:56:37,215 INFO [RS:0;jenkins-hbase4:45193] regionserver.HeapMemoryManager(220): Stopping 2023-03-30 16:56:37,216 INFO [RS:0;jenkins-hbase4:45193] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-30 16:56:37,216 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-30 16:56:37,216 INFO [RS:0;jenkins-hbase4:45193] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-30 16:56:37,216 INFO [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(3304): Received CLOSE for e324c540a5e4b217db9fa3051ef755ac 2023-03-30 16:56:37,217 INFO [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:37,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e324c540a5e4b217db9fa3051ef755ac, disabling compactions & flushes 2023-03-30 16:56:37,217 DEBUG [RS:0;jenkins-hbase4:45193] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e714ffa to 127.0.0.1:58435 2023-03-30 16:56:37,217 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac. 2023-03-30 16:56:37,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac. 2023-03-30 16:56:37,217 DEBUG [RS:0;jenkins-hbase4:45193] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:56:37,218 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac. after waiting 0 ms 2023-03-30 16:56:37,218 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac. 2023-03-30 16:56:37,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing e324c540a5e4b217db9fa3051ef755ac 1/1 column families, dataSize=78 B heapSize=488 B 2023-03-30 16:56:37,218 INFO [RS:0;jenkins-hbase4:45193] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-30 16:56:37,218 INFO [RS:0;jenkins-hbase4:45193] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-30 16:56:37,218 INFO [RS:0;jenkins-hbase4:45193] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-30 16:56:37,218 INFO [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-03-30 16:56:37,218 INFO [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-03-30 16:56:37,219 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-30 16:56:37,219 DEBUG [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(1479): Online Regions={e324c540a5e4b217db9fa3051ef755ac=hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac., 1588230740=hbase:meta,,1.1588230740} 2023-03-30 16:56:37,219 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-30 16:56:37,219 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-30 16:56:37,219 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-30 16:56:37,219 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-30 16:56:37,219 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-03-30 16:56:37,221 DEBUG [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(1505): Waiting on 1588230740, e324c540a5e4b217db9fa3051ef755ac 2023-03-30 16:56:37,233 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/.tmp/info/83be3985e3ad4787a868afc49a05f279 2023-03-30 16:56:37,237 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 83be3985e3ad4787a868afc49a05f279 2023-03-30 16:56:37,245 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/.tmp/rep_barrier/ca4f436cc81e4c6592f0d72bae7dc726 2023-03-30 16:56:37,249 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ca4f436cc81e4c6592f0d72bae7dc726 2023-03-30 16:56:37,303 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-30 16:56:37,421 DEBUG [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(1505): Waiting on 1588230740, e324c540a5e4b217db9fa3051ef755ac 2023-03-30 16:56:37,574 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-03-30 16:56:37,574 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-03-30 16:56:37,621 DEBUG [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(1505): Waiting on 1588230740, e324c540a5e4b217db9fa3051ef755ac 2023-03-30 16:56:37,633 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/namespace/e324c540a5e4b217db9fa3051ef755ac/.tmp/info/0d16b8b5ca51497ebe33530f03a77edf 2023-03-30 16:56:37,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/namespace/e324c540a5e4b217db9fa3051ef755ac/.tmp/info/0d16b8b5ca51497ebe33530f03a77edf as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/namespace/e324c540a5e4b217db9fa3051ef755ac/info/0d16b8b5ca51497ebe33530f03a77edf 2023-03-30 16:56:37,640 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/namespace/e324c540a5e4b217db9fa3051ef755ac/info/0d16b8b5ca51497ebe33530f03a77edf, entries=2, sequenceid=6, filesize=4.8 K 2023-03-30 16:56:37,641 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for e324c540a5e4b217db9fa3051ef755ac in 423ms, sequenceid=6, compaction requested=false 2023-03-30 16:56:37,646 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/namespace/e324c540a5e4b217db9fa3051ef755ac/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-03-30 16:56:37,647 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac. 2023-03-30 16:56:37,647 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e324c540a5e4b217db9fa3051ef755ac: 2023-03-30 16:56:37,647 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680195216892.e324c540a5e4b217db9fa3051ef755ac. 2023-03-30 16:56:37,654 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/.tmp/table/08bd786cf97544e190922d82e641451a 2023-03-30 16:56:37,657 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 08bd786cf97544e190922d82e641451a 2023-03-30 16:56:37,658 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/.tmp/info/83be3985e3ad4787a868afc49a05f279 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/info/83be3985e3ad4787a868afc49a05f279 2023-03-30 16:56:37,660 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 83be3985e3ad4787a868afc49a05f279 2023-03-30 16:56:37,661 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/info/83be3985e3ad4787a868afc49a05f279, entries=22, sequenceid=93, filesize=7.1 K 2023-03-30 16:56:37,662 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/.tmp/rep_barrier/ca4f436cc81e4c6592f0d72bae7dc726 as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/rep_barrier/ca4f436cc81e4c6592f0d72bae7dc726 2023-03-30 16:56:37,665 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ca4f436cc81e4c6592f0d72bae7dc726 2023-03-30 16:56:37,665 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/rep_barrier/ca4f436cc81e4c6592f0d72bae7dc726, entries=6, sequenceid=93, filesize=5.5 K 2023-03-30 16:56:37,665 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/.tmp/table/08bd786cf97544e190922d82e641451a as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/table/08bd786cf97544e190922d82e641451a 2023-03-30 16:56:37,668 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 08bd786cf97544e190922d82e641451a 2023-03-30 16:56:37,668 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/table/08bd786cf97544e190922d82e641451a, entries=9, sequenceid=93, filesize=5.5 K 2023-03-30 16:56:37,668 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 449ms, sequenceid=93, compaction requested=false 2023-03-30 16:56:37,675 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-03-30 16:56:37,676 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-30 16:56:37,677 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-30 16:56:37,677 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-30 16:56:37,677 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-03-30 16:56:37,821 INFO [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,45193,1680195214285; all regions closed. 2023-03-30 16:56:37,830 DEBUG [RS:0;jenkins-hbase4:45193] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/oldWALs 2023-03-30 16:56:37,830 INFO [RS:0;jenkins-hbase4:45193] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C45193%2C1680195214285.meta:.meta(num 1680195216641) 2023-03-30 16:56:37,834 DEBUG [RS:0;jenkins-hbase4:45193] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/oldWALs 2023-03-30 16:56:37,834 INFO [RS:0;jenkins-hbase4:45193] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C45193%2C1680195214285:(num 1680195215722) 2023-03-30 16:56:37,834 DEBUG [RS:0;jenkins-hbase4:45193] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:56:37,834 INFO [RS:0;jenkins-hbase4:45193] regionserver.LeaseManager(133): Closed leases 2023-03-30 16:56:37,834 INFO [RS:0;jenkins-hbase4:45193] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-03-30 16:56:37,835 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-30 16:56:37,836 INFO [RS:0;jenkins-hbase4:45193] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:45193 2023-03-30 16:56:37,842 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,45193,1680195214285 2023-03-30 16:56:37,842 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-30 16:56:37,842 ERROR [Listener at localhost/38075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@60deb4a7 rejected from java.util.concurrent.ThreadPoolExecutor@64e12c64[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2390] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-30 16:56:37,843 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-30 16:56:37,843 ERROR [Listener at localhost/38075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5666c53 rejected from java.util.concurrent.ThreadPoolExecutor@64e12c64[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2390] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-30 16:56:37,843 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,45193,1680195214285] 2023-03-30 16:56:37,844 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,45193,1680195214285; numProcessing=1 2023-03-30 16:56:37,845 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,45193,1680195214285 already deleted, retry=false 2023-03-30 16:56:37,845 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,45193,1680195214285 expired; onlineServers=0 2023-03-30 16:56:37,845 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,40565,1680195213071' ***** 2023-03-30 16:56:37,845 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-03-30 16:56:37,845 DEBUG [M:0;jenkins-hbase4:40565] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a308c86, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-30 16:56:37,845 INFO [M:0;jenkins-hbase4:40565] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,40565,1680195213071 2023-03-30 16:56:37,845 INFO [M:0;jenkins-hbase4:40565] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,40565,1680195213071; all regions closed. 2023-03-30 16:56:37,846 DEBUG [M:0;jenkins-hbase4:40565] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-30 16:56:37,846 DEBUG [M:0;jenkins-hbase4:40565] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-03-30 16:56:37,846 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-03-30 16:56:37,846 DEBUG [M:0;jenkins-hbase4:40565] cleaner.HFileCleaner(317): Stopping file delete threads 2023-03-30 16:56:37,846 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680195215366] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680195215366,5,FailOnTimeoutGroup] 2023-03-30 16:56:37,846 INFO [M:0;jenkins-hbase4:40565] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-03-30 16:56:37,846 INFO [M:0;jenkins-hbase4:40565] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-03-30 16:56:37,846 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680195215367] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680195215367,5,FailOnTimeoutGroup] 2023-03-30 16:56:37,846 INFO [M:0;jenkins-hbase4:40565] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-03-30 16:56:37,847 DEBUG [M:0;jenkins-hbase4:40565] master.HMaster(1512): Stopping service threads 2023-03-30 16:56:37,847 INFO [M:0;jenkins-hbase4:40565] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-03-30 16:56:37,847 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-03-30 16:56:37,847 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-30 16:56:37,847 ERROR [M:0;jenkins-hbase4:40565] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-03-30 16:56:37,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-30 16:56:37,847 INFO [M:0;jenkins-hbase4:40565] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-03-30 16:56:37,848 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-03-30 16:56:37,848 DEBUG [M:0;jenkins-hbase4:40565] zookeeper.ZKUtil(398): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-03-30 16:56:37,848 WARN [M:0;jenkins-hbase4:40565] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-03-30 16:56:37,848 INFO [M:0;jenkins-hbase4:40565] assignment.AssignmentManager(315): Stopping assignment manager 2023-03-30 16:56:37,848 INFO [M:0;jenkins-hbase4:40565] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-03-30 16:56:37,849 DEBUG [M:0;jenkins-hbase4:40565] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-30 16:56:37,849 INFO [M:0;jenkins-hbase4:40565] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-30 16:56:37,849 DEBUG [M:0;jenkins-hbase4:40565] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-30 16:56:37,849 DEBUG [M:0;jenkins-hbase4:40565] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-30 16:56:37,849 DEBUG [M:0;jenkins-hbase4:40565] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-30 16:56:37,849 INFO [M:0;jenkins-hbase4:40565] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.86 KB heapSize=393.70 KB 2023-03-30 16:56:37,858 INFO [M:0;jenkins-hbase4:40565] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.86 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/9d853c23a97f44169bb08f0946cc68cc 2023-03-30 16:56:37,862 INFO [M:0;jenkins-hbase4:40565] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9d853c23a97f44169bb08f0946cc68cc 2023-03-30 16:56:37,862 DEBUG [M:0;jenkins-hbase4:40565] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/9d853c23a97f44169bb08f0946cc68cc as hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/9d853c23a97f44169bb08f0946cc68cc 2023-03-30 16:56:37,865 INFO [M:0;jenkins-hbase4:40565] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9d853c23a97f44169bb08f0946cc68cc 2023-03-30 16:56:37,865 INFO [M:0;jenkins-hbase4:40565] regionserver.HStore(1080): Added hdfs://localhost:46323/user/jenkins/test-data/de785e70-db28-b9ae-fabb-25fdbdcd857c/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/9d853c23a97f44169bb08f0946cc68cc, entries=65, sequenceid=579, filesize=24.6 K 2023-03-30 16:56:37,866 INFO [M:0;jenkins-hbase4:40565] regionserver.HRegion(2948): Finished flush of dataSize ~344.86 KB/353136, heapSize ~393.69 KB/403136, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 17ms, sequenceid=579, compaction requested=false 2023-03-30 16:56:37,867 INFO [M:0;jenkins-hbase4:40565] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-30 16:56:37,867 DEBUG [M:0;jenkins-hbase4:40565] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-30 16:56:37,870 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-30 16:56:37,870 INFO [M:0;jenkins-hbase4:40565] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-03-30 16:56:37,871 INFO [M:0;jenkins-hbase4:40565] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40565 2023-03-30 16:56:37,873 DEBUG [M:0;jenkins-hbase4:40565] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,40565,1680195213071 already deleted, retry=false 2023-03-30 16:56:37,945 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-30 16:56:37,945 INFO [RS:0;jenkins-hbase4:45193] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,45193,1680195214285; zookeeper connection closed. 2023-03-30 16:56:37,945 ERROR [Listener at localhost/38075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7d6717bd rejected from java.util.concurrent.ThreadPoolExecutor@64e12c64[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2390] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-30 16:56:37,946 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): regionserver:45193-0x10136583d700001, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-30 16:56:37,946 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@20bbb9bf] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@20bbb9bf 2023-03-30 16:56:37,946 ERROR [Listener at localhost/38075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@77feb94d rejected from java.util.concurrent.ThreadPoolExecutor@64e12c64[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2390] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-30 16:56:37,946 INFO [Listener at localhost/38075] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-03-30 16:56:38,045 INFO [M:0;jenkins-hbase4:40565] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,40565,1680195213071; zookeeper connection closed. 2023-03-30 16:56:38,045 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-30 16:56:38,045 ERROR [Listener at localhost/38075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@63797713 rejected from java.util.concurrent.ThreadPoolExecutor@69ea4d01[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3991] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-30 16:56:38,045 DEBUG [Listener at localhost/38075-EventThread] zookeeper.ZKWatcher(600): master:40565-0x10136583d700000, quorum=127.0.0.1:58435, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-30 16:56:38,045 ERROR [Listener at localhost/38075-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@61bebe6b rejected from java.util.concurrent.ThreadPoolExecutor@69ea4d01[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3991] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-30 16:56:38,046 WARN [Listener at localhost/38075] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-30 16:56:38,050 INFO [Listener at localhost/38075] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-30 16:56:38,065 WARN [BP-917933237-172.31.14.131-1680195209756 heartbeating to localhost/127.0.0.1:46323] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-917933237-172.31.14.131-1680195209756 (Datanode Uuid feae19bc-dd98-4b85-9889-0ea49bd6ef8b) service to localhost/127.0.0.1:46323 2023-03-30 16:56:38,068 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/cluster_6981959d-287c-2be4-2da0-db5747864972/dfs/data/data1/current/BP-917933237-172.31.14.131-1680195209756] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-30 16:56:38,068 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/781b4dcd-f257-6713-5ff5-3faa65c57cea/cluster_6981959d-287c-2be4-2da0-db5747864972/dfs/data/data2/current/BP-917933237-172.31.14.131-1680195209756] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-30 16:56:38,190 INFO [Listener at localhost/38075] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-30 16:56:38,354 INFO [Listener at localhost/38075] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-03-30 16:56:38,519 INFO [Listener at localhost/38075] hbase.HBaseTestingUtility(1293): Minicluster is down